From 8b8788189a784d4dced339d6dd5dafe84e019396 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Fri, 8 Aug 2025 12:52:27 -0400 Subject: [PATCH 1/7] Initial revision of PEP 797. --- .github/CODEOWNERS | 1 + peps/pep-0797.rst | 270 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 271 insertions(+) create mode 100644 peps/pep-0797.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 1c58bc21620..47b1f46246f 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -673,6 +673,7 @@ peps/pep-0791.rst @vstinner peps/pep-0792.rst @dstufft peps/pep-0793.rst @encukou peps/pep-0794.rst @brettcannon +peps/pep-0797.rst @ZeroIntensity peps/pep-0798.rst @JelleZijlstra peps/pep-0799.rst @pablogsal peps/pep-0800.rst @JelleZijlstra diff --git a/peps/pep-0797.rst b/peps/pep-0797.rst new file mode 100644 index 00000000000..c5257ace44f --- /dev/null +++ b/peps/pep-0797.rst @@ -0,0 +1,270 @@ +PEP: 797 +Title: Shared Object Proxies for Subinterpreters +Author: Peter Bierma +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 08-Aug-2025 +Python-Version: 3.15 +Post-History: `01-Jul-2025 `__ + + +Abstract +======== + +This PEP introduces a new :func:`~concurrent.interpreters.share` function to +the :mod:`concurrent.interpreters` module, which allows *any* arbitrary object +to be shared for a period of time. + +For example:: + + from concurrent import interpreters + + with open("spanish_inquisition.txt") as unshareable: + interp = interpreters.create() + with interpreters.share(unshareable) as proxy: + interp.prepare_main(file=proxy) + interp.exec("file.write('I didn't expect the Spanish Inquisition')") + + +Motivation +========== + +Many Objects Cannot be Shared Between Subinterpreters +----------------------------------------------------- + +In Python 3.14, the new :mod:`concurrent.interpreters` module can be used to +create multiple interpreters in a single Python process. This works well for +stateless code (that is, code that doesn't need any external data) and objects +that can be serialized, but it is fairly common for applications to want to use +highly-complex data structures (that cannot be serialized) with their +concurrency. + +Currently, :mod:`!concurrent.interpreters` can only share +:ref:`a handful of types ` natively, and then falls back +to the :mod:`pickle` module for other types. This can be very limited, as many +types of objects cannot be pickled. + +Rationale +========= + +A Fallback for Object Sharing +----------------------------- + +A shared object proxy is designed to be a fallback for sharing an object +between interpreters, because it's generally slow and causes increased memory +usage (due to :term:`immortality `, which will be discussed more +later). As such, this PEP does not make other mechanisms for sharing objects +(namely, serialization) obsolete. A shared object proxy should only be used as +a last-resort silver bullet for highly complex objects that cannot be +serialized or shared in any other way. + +Specification +============= + +The ``SharedObjectProxy`` Type +------------------------------ + +.. class:: concurrent.interpreters.SharedObjectProxy + + A proxy type that allows thread-safe access to an object across multiple + interpreters. This cannot be constructed from Python; instead, use the + :func:`~concurrent.interpreters.share` function. + + When interacting with the wrapped object, the proxy will switch to the + interpreter in which the object was created. Arguments passed to anything + on the proxy are also wrapped in a new shared object proxy if the type + isn't natively shareable (so, for example, strings would not be wrapped + in an object proxy, but file objects would). The same goes for return + values. + + For thread-safety purposes, an instance of ``SharedObjectProxy`` is + always :term:`immortal`. This means that it won't be deallocated for the + lifetime of the interpreter. When an object proxy is done being used, it + clears its reference to the object that it wraps and allows itself to be + reused. This prevents extreme memory accumulation. + + In addition, all object proxies have an implicit context that manages them. + This context is determined by the most recent call to + :func:`~concurrent.interpreters.share` in the current thread. When the context + finishes, all object proxies created under that context are cleared, allowing + them to be reused in a new context. + +Thread State Switching +********************** + +At the C level, all objects in Python's C API are interacted with through their +type (a pointer to a :c:type:`PyTypeObject`). For example, to call an object, +the interpreter will access the :c:member:`~PyTypeObject.tp_call` field on the +object's type. This is where the magic of a shared object proxy can happen. + +The :c:type:`!PyTypeObject` for a shared object proxy must be such a type that +implements wrapping behavior for every single field on the type object +structure. So, going back to ``tp_call``, an object proxy must be able to +"intercept" the call in such a way where the wrapped object's ``tp_call`` +slot can be executed without thread-safety issues. This is done by switching +the :term:`attached thread state`. + +In the C API, a :term:`thread state` belongs to a certain interpreter, and by +holding an attached thread state, the thread may interact with any object +belonging to its interpreter. This is because holding an attached thread state +implies things like holding the :term:`GIL`, which make object access thread-safe. + +.. note:: + + On the :term:`free threaded ` build, it is still required + to hold an :term:`attached thread state` to interact with objects in the + C API. + +So, with that in mind, the only thing that the object proxy has to do to call +a type slot is hold an attached thread state for the object's interpreter. +This is the fundamental idea of how a shared object proxy works: allow access +from any interpreter, but switch to one the wrapped object needs when a type +slot is called. + +Sharing Arguments and Return Values +*********************************** + +Once the attached thread state has been switched to match a wrapped object's +interpreter, passed arguments and the return value of the slot need to be shared +back to the caller. This is done by first attempting to share them natively +(for example, with objects such as ``True`` or ``False``), and then falling +back to creating a new shared object proxy if all else fails. The new proxy +is given the same context as the current proxy, meaning the newly wrapped object +will be able to be freed once the :func:`~concurrent.interpreters.share` context +is closed. + +The Sharing APIs +---------------- + +.. function:: concurrent.interpreters.share(obj) + + Wrap *obj* in a :class:`~concurrent.interpreters.SharedObjectProxy`, + allowing it to be used in other interpreter APIs as if it were natively shareable. + + This returns a :term:`context manager`. The resulting object is the proxy + that can be shared (meaning that *obj* is left unchanged). After the context + is closed, the proxy will release its reference to *obj* and allow itself to + be reused for a future call to ``share``. + + If this function is used on an existing shared object proxy, it is assigned + a new context, preventing it from being cleared when the parent ``share`` + context finishes. + + For example: + + .. code-block:: python + + from concurrent import interpreters + + with open("spanish_inquisition.txt") as unshareable: + interp = interpreters.create() + with interpreters.share(unshareable) as proxy: + interp.prepare_main(file=proxy) + interp.exec("file.write('I didn't expect the Spanish Inquisition')") + + + .. note:: + + ``None`` cannot be used with this function, as ``None`` is a special + value reserved for dead object proxies. Since ``None`` is natively + shareable, there's no need to pass it to this function anyway. + +.. function:: concurrent.interpreters.share_forever(obj) + + Similar to :func:`~concurrent.interpreters.share`, but *does not* give the resulting + proxy a context, meaning it will live forever (unless a call to ``share`` + explicitly gives the proxy a new lifetime). As such, this function does not + return a :term:`context manager`. + + For example: + + .. code-block:: python + + from concurrent import interpreters + + with open("spanish_inquisition.txt") as unshareable: + interp = interpreters.create() + proxy = interpreters.share_forever(unshareable) + interp.prepare_main(file=proxy) + # Note: the bound method object for file.write() will also live + # forever in a proxy. + interp.exec("file.write('I didn't expect the Spanish Inquisition')") + + .. warning:: + + Proxies created as a result of the returned proxy (for example, bound + method objects) will also exist for the lifetime of the interpreter, + which can lead to high memory usage. + + +Multithreaded Scaling +--------------------- + +Since an object proxy mostly interacts with an object normally, there shouldn't +be much additional overhead on using the object once the thread state has been +switched. However, this means that when the :term:`GIL` is enabled, you may lose +some of the concurrency benefits from subinterpreters, because threads will be +stuck waiting on the GIL for a wrapped object. + +Backwards Compatibility +======================= + +In order to implement the immortality mechanism used by shared object proxies, +several assumptions had to be made about the object lifecycle in the C API. +So, some best practices in the C API (such as using the object allocator for +objects) are made harder requirements by the implementation of this PEP. + +The author of this PEP believes it is unlikely that this will cause breakage, +as he has not ever seen code in the wild that violates the assumptions made +about the object lifecycle as required by the reference implementation. + +Security Implications +===================== + +The largest issue with shared object proxies is that in order to have +thread-safe reference counting operations, they must be :term:`immortal`, +which prevents any concurrent modification to their reference count. +This can cause them to take up very large amounts of memory if mismanaged. + +The :func:`~concurrent.interpreters.share` context manager does its best +to avoid this issue by manually clearing references at the end of an object +proxy's usage (allowing mortal objects to be freed), as well as avoiding +the allocation of new object proxies by reusing dead ones (that is, object +proxies with a cleared reference). + +How to Teach This +================= + +New APIs and important information about how to use them will be added to the +:mod:`concurrent.interpreters` documentation. An informational PEP regarding +the new immortality mechanisms included in the reference implementation will +be written if this PEP is accepted. + +Reference Implementation +======================== + +The reference implementation of this PEP can be found +`here `_. + +Rejected Ideas +============== + +Why Not Atomic Reference Counting? +---------------------------------- + +Immortality seems to be the driver for a lot of complexity in this proposal; +why not use atomic reference counting instead? + +Atomic reference counting has been tried before in previous :term:`GIL` +removal attempts, but unfortunately added too much overhead to CPython to be +feasible, because atomic "add" operations are much slower than their non-atomic +counterparts. Immortality, while complex, has the benefit of being efficient +and thread-safe without needing to slow down single-threaded performance with +reference counting. + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From d1e458fd3519c145f6fe1a803c0fd6492af3657b Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 15 Sep 2025 08:13:42 -0400 Subject: [PATCH 2/7] General clarity improvements. --- peps/pep-0797.rst | 51 ++++++++++++++++++++++++----------------------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/peps/pep-0797.rst b/peps/pep-0797.rst index c5257ace44f..8b96ccc0720 100644 --- a/peps/pep-0797.rst +++ b/peps/pep-0797.rst @@ -35,15 +35,16 @@ Many Objects Cannot be Shared Between Subinterpreters In Python 3.14, the new :mod:`concurrent.interpreters` module can be used to create multiple interpreters in a single Python process. This works well for -stateless code (that is, code that doesn't need any external data) and objects -that can be serialized, but it is fairly common for applications to want to use -highly-complex data structures (that cannot be serialized) with their -concurrency. +stateless code (that is, code that doesn't need any state from a caller) and +objects that can be serialized, but it is fairly common for applications to +want to use highly-complex data structures (that cannot be serialized) with +their concurrency. Currently, :mod:`!concurrent.interpreters` can only share :ref:`a handful of types ` natively, and then falls back to the :mod:`pickle` module for other types. This can be very limited, as many -types of objects cannot be pickled. +types of objects cannot be pickled. For example, file objects returned by +:func:`open` cannot be serialized through ``pickle``. Rationale ========= @@ -56,8 +57,8 @@ between interpreters, because it's generally slow and causes increased memory usage (due to :term:`immortality `, which will be discussed more later). As such, this PEP does not make other mechanisms for sharing objects (namely, serialization) obsolete. A shared object proxy should only be used as -a last-resort silver bullet for highly complex objects that cannot be -serialized or shared in any other way. +a last-resort for highly complex objects that cannot be serialized or shared +in any other way. Specification ============= @@ -98,12 +99,12 @@ type (a pointer to a :c:type:`PyTypeObject`). For example, to call an object, the interpreter will access the :c:member:`~PyTypeObject.tp_call` field on the object's type. This is where the magic of a shared object proxy can happen. -The :c:type:`!PyTypeObject` for a shared object proxy must be such a type that -implements wrapping behavior for every single field on the type object -structure. So, going back to ``tp_call``, an object proxy must be able to -"intercept" the call in such a way where the wrapped object's ``tp_call`` -slot can be executed without thread-safety issues. This is done by switching -the :term:`attached thread state`. +The :c:type:`!PyTypeObject` for a shared object proxy must implement +wrapping behavior for every single field on the type object structure. +So, going back to ``tp_call``, an object proxy must be able to "intercept" the +call in such a way where the wrapped object's ``tp_call`` slot can be executed +without thread-safety issues. This is done by switching the +:term:`attached thread state`. In the C API, a :term:`thread state` belongs to a certain interpreter, and by holding an attached thread state, the thread may interact with any object @@ -119,19 +120,19 @@ implies things like holding the :term:`GIL`, which make object access thread-saf So, with that in mind, the only thing that the object proxy has to do to call a type slot is hold an attached thread state for the object's interpreter. This is the fundamental idea of how a shared object proxy works: allow access -from any interpreter, but switch to one the wrapped object needs when a type +from any interpreter, but switch to the wrapped object's interpreter when a type slot is called. Sharing Arguments and Return Values *********************************** Once the attached thread state has been switched to match a wrapped object's -interpreter, passed arguments and the return value of the slot need to be shared -back to the caller. This is done by first attempting to share them natively -(for example, with objects such as ``True`` or ``False``), and then falling -back to creating a new shared object proxy if all else fails. The new proxy -is given the same context as the current proxy, meaning the newly wrapped object -will be able to be freed once the :func:`~concurrent.interpreters.share` context +interpreter, arguments and the return value (if it's a ``PyObject *``) of the +slot need to be shared back to the caller. This is done by first attempting to +share them natively (for example, with ``pickle``), and then falling back to +creating a new shared object proxy if all else fails. The new proxy is given +the same context as the current proxy, meaning the newly wrapped object will +be able to be freed once the :func:`~concurrent.interpreters.share` context is closed. The Sharing APIs @@ -142,10 +143,10 @@ The Sharing APIs Wrap *obj* in a :class:`~concurrent.interpreters.SharedObjectProxy`, allowing it to be used in other interpreter APIs as if it were natively shareable. - This returns a :term:`context manager`. The resulting object is the proxy - that can be shared (meaning that *obj* is left unchanged). After the context - is closed, the proxy will release its reference to *obj* and allow itself to - be reused for a future call to ``share``. + This returns a :term:`context manager`. The resulting object from the + context is the proxy that can be shared. After the context is closed, the + proxy will release its reference to *obj* and allow itself to be reused + for a future call to ``share``. If this function is used on an existing shared object proxy, it is assigned a new context, preventing it from being cleared when the parent ``share`` @@ -205,7 +206,7 @@ Since an object proxy mostly interacts with an object normally, there shouldn't be much additional overhead on using the object once the thread state has been switched. However, this means that when the :term:`GIL` is enabled, you may lose some of the concurrency benefits from subinterpreters, because threads will be -stuck waiting on the GIL for a wrapped object. +stuck waiting on the GIL of a wrapped object's interpreter. Backwards Compatibility ======================= From c2b9e688ea48387924d342d31595a9c1e0cf2e51 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Wed, 26 Nov 2025 22:48:30 -0500 Subject: [PATCH 3/7] Remove immortality from the proposal. --- peps/pep-0797.rst | 225 ++++++++++++++-------------------------------- 1 file changed, 68 insertions(+), 157 deletions(-) diff --git a/peps/pep-0797.rst b/peps/pep-0797.rst index 8b96ccc0720..3e9f1f1a3f3 100644 --- a/peps/pep-0797.rst +++ b/peps/pep-0797.rst @@ -1,6 +1,6 @@ PEP: 797 Title: Shared Object Proxies for Subinterpreters -Author: Peter Bierma +Author: Peter Bierma Discussions-To: Pending Status: Draft Type: Standards Track @@ -14,7 +14,8 @@ Abstract This PEP introduces a new :func:`~concurrent.interpreters.share` function to the :mod:`concurrent.interpreters` module, which allows *any* arbitrary object -to be shared for a period of time. +to be shared across multiple interpreters, at the cost of being less efficient +under multithreaded code while accessing it. For example:: @@ -22,78 +23,14 @@ For example:: with open("spanish_inquisition.txt") as unshareable: interp = interpreters.create() - with interpreters.share(unshareable) as proxy: - interp.prepare_main(file=proxy) - interp.exec("file.write('I didn't expect the Spanish Inquisition')") + proxy = interpreters.share(unshareable) as proxy: + interp.prepare_main(file=proxy) + interp.exec("file.write('I didn't expect the Spanish Inquisition')") -Motivation +Background ========== -Many Objects Cannot be Shared Between Subinterpreters ------------------------------------------------------ - -In Python 3.14, the new :mod:`concurrent.interpreters` module can be used to -create multiple interpreters in a single Python process. This works well for -stateless code (that is, code that doesn't need any state from a caller) and -objects that can be serialized, but it is fairly common for applications to -want to use highly-complex data structures (that cannot be serialized) with -their concurrency. - -Currently, :mod:`!concurrent.interpreters` can only share -:ref:`a handful of types ` natively, and then falls back -to the :mod:`pickle` module for other types. This can be very limited, as many -types of objects cannot be pickled. For example, file objects returned by -:func:`open` cannot be serialized through ``pickle``. - -Rationale -========= - -A Fallback for Object Sharing ------------------------------ - -A shared object proxy is designed to be a fallback for sharing an object -between interpreters, because it's generally slow and causes increased memory -usage (due to :term:`immortality `, which will be discussed more -later). As such, this PEP does not make other mechanisms for sharing objects -(namely, serialization) obsolete. A shared object proxy should only be used as -a last-resort for highly complex objects that cannot be serialized or shared -in any other way. - -Specification -============= - -The ``SharedObjectProxy`` Type ------------------------------- - -.. class:: concurrent.interpreters.SharedObjectProxy - - A proxy type that allows thread-safe access to an object across multiple - interpreters. This cannot be constructed from Python; instead, use the - :func:`~concurrent.interpreters.share` function. - - When interacting with the wrapped object, the proxy will switch to the - interpreter in which the object was created. Arguments passed to anything - on the proxy are also wrapped in a new shared object proxy if the type - isn't natively shareable (so, for example, strings would not be wrapped - in an object proxy, but file objects would). The same goes for return - values. - - For thread-safety purposes, an instance of ``SharedObjectProxy`` is - always :term:`immortal`. This means that it won't be deallocated for the - lifetime of the interpreter. When an object proxy is done being used, it - clears its reference to the object that it wraps and allows itself to be - reused. This prevents extreme memory accumulation. - - In addition, all object proxies have an implicit context that manages them. - This context is determined by the most recent call to - :func:`~concurrent.interpreters.share` in the current thread. When the context - finishes, all object proxies created under that context are cleared, allowing - them to be reused in a new context. - -Thread State Switching -********************** - At the C level, all objects in Python's C API are interacted with through their type (a pointer to a :c:type:`PyTypeObject`). For example, to call an object, the interpreter will access the :c:member:`~PyTypeObject.tp_call` field on the @@ -111,21 +48,12 @@ holding an attached thread state, the thread may interact with any object belonging to its interpreter. This is because holding an attached thread state implies things like holding the :term:`GIL`, which make object access thread-safe. -.. note:: - - On the :term:`free threaded ` build, it is still required - to hold an :term:`attached thread state` to interact with objects in the - C API. - So, with that in mind, the only thing that the object proxy has to do to call a type slot is hold an attached thread state for the object's interpreter. This is the fundamental idea of how a shared object proxy works: allow access from any interpreter, but switch to the wrapped object's interpreter when a type slot is called. -Sharing Arguments and Return Values -*********************************** - Once the attached thread state has been switched to match a wrapped object's interpreter, arguments and the return value (if it's a ``PyObject *``) of the slot need to be shared back to the caller. This is done by first attempting to @@ -135,48 +63,61 @@ the same context as the current proxy, meaning the newly wrapped object will be able to be freed once the :func:`~concurrent.interpreters.share` context is closed. -The Sharing APIs ----------------- +Motivation +========== -.. function:: concurrent.interpreters.share(obj) +Many Objects Cannot be Shared Between Subinterpreters +----------------------------------------------------- - Wrap *obj* in a :class:`~concurrent.interpreters.SharedObjectProxy`, - allowing it to be used in other interpreter APIs as if it were natively shareable. +In Python 3.14, the new :mod:`concurrent.interpreters` module can be used to +create multiple interpreters in a single Python process. This works well for +stateless code (that is, code that doesn't need any state from a caller) and +objects that can be serialized, but it is fairly common for applications to +want to use highly-complex data structures (that cannot be serialized) with +their concurrency. - This returns a :term:`context manager`. The resulting object from the - context is the proxy that can be shared. After the context is closed, the - proxy will release its reference to *obj* and allow itself to be reused - for a future call to ``share``. +Currently, :mod:`!concurrent.interpreters` can only share +:ref:`a handful of types ` natively, and then falls back +to the :mod:`pickle` module for other types. This can be very limited, as many +types of objects cannot be pickled. For example, file objects returned by +:func:`open` cannot be serialized through ``pickle``. - If this function is used on an existing shared object proxy, it is assigned - a new context, preventing it from being cleared when the parent ``share`` - context finishes. +Rationale +========= - For example: +A Fallback for Object Sharing +----------------------------- - .. code-block:: python +A shared object proxy is designed to be a fallback for sharing an object +between interpreters, because it does not scale well under multiple threads. +As such, this PEP does not make other mechanisms for sharing objects +(namely, serialization) obsolete. A shared object proxy should only be used as +a last-resort for highly complex objects that cannot be serialized or shared +in any other way. - from concurrent import interpreters +Specification +============= - with open("spanish_inquisition.txt") as unshareable: - interp = interpreters.create() - with interpreters.share(unshareable) as proxy: - interp.prepare_main(file=proxy) - interp.exec("file.write('I didn't expect the Spanish Inquisition')") +.. class:: concurrent.interpreters.SharedObjectProxy + A proxy type that allows access to an object across multiple interpreters. + This cannot be constructed from Python; instead, use the + :func:`~concurrent.interpreters.share` function. - .. note:: + When interacting with the wrapped object, the proxy will switch to the + interpreter in which the object was created. Arguments passed to anything + on the proxy are also wrapped in a new object proxy if the type isn't + natively shareable (so, for example, strings would not be wrapped in an + object proxy, but file objects would). The same goes for return values. - ``None`` cannot be used with this function, as ``None`` is a special - value reserved for dead object proxies. Since ``None`` is natively - shareable, there's no need to pass it to this function anyway. + An object proxy does not scale well under multiple threads; there will + be contention on locks and the :term:`GIL` if it is enabled. -.. function:: concurrent.interpreters.share_forever(obj) - Similar to :func:`~concurrent.interpreters.share`, but *does not* give the resulting - proxy a context, meaning it will live forever (unless a call to ``share`` - explicitly gives the proxy a new lifetime). As such, this function does not - return a :term:`context manager`. +.. function:: concurrent.interpreters.share(obj) + + Wrap *obj* in a :class:`~concurrent.interpreters.SharedObjectProxy`, + allowing it to be used in other interpreter APIs as if it were natively shareable. For example: @@ -186,61 +127,26 @@ The Sharing APIs with open("spanish_inquisition.txt") as unshareable: interp = interpreters.create() - proxy = interpreters.share_forever(unshareable) + proxy = interpreters.share(unshareable) interp.prepare_main(file=proxy) - # Note: the bound method object for file.write() will also live - # forever in a proxy. interp.exec("file.write('I didn't expect the Spanish Inquisition')") - .. warning:: - - Proxies created as a result of the returned proxy (for example, bound - method objects) will also exist for the lifetime of the interpreter, - which can lead to high memory usage. - - -Multithreaded Scaling ---------------------- - -Since an object proxy mostly interacts with an object normally, there shouldn't -be much additional overhead on using the object once the thread state has been -switched. However, this means that when the :term:`GIL` is enabled, you may lose -some of the concurrency benefits from subinterpreters, because threads will be -stuck waiting on the GIL of a wrapped object's interpreter. Backwards Compatibility ======================= -In order to implement the immortality mechanism used by shared object proxies, -several assumptions had to be made about the object lifecycle in the C API. -So, some best practices in the C API (such as using the object allocator for -objects) are made harder requirements by the implementation of this PEP. - -The author of this PEP believes it is unlikely that this will cause breakage, -as he has not ever seen code in the wild that violates the assumptions made -about the object lifecycle as required by the reference implementation. +This PEP has no known backwards compatibility issues. Security Implications ===================== -The largest issue with shared object proxies is that in order to have -thread-safe reference counting operations, they must be :term:`immortal`, -which prevents any concurrent modification to their reference count. -This can cause them to take up very large amounts of memory if mismanaged. - -The :func:`~concurrent.interpreters.share` context manager does its best -to avoid this issue by manually clearing references at the end of an object -proxy's usage (allowing mortal objects to be freed), as well as avoiding -the allocation of new object proxies by reusing dead ones (that is, object -proxies with a cleared reference). +This PEP has no known backwards security implications. How to Teach This ================= New APIs and important information about how to use them will be added to the -:mod:`concurrent.interpreters` documentation. An informational PEP regarding -the new immortality mechanisms included in the reference implementation will -be written if this PEP is accepted. +:mod:`concurrent.interpreters` documentation. Reference Implementation ======================== @@ -251,18 +157,23 @@ The reference implementation of this PEP can be found Rejected Ideas ============== -Why Not Atomic Reference Counting? ----------------------------------- +Directly Sharing Proxy Objects +------------------------------ + +The initial revision of this proposal took an approach where an instance of +:class:`~conccurent.interpreters.SharedObjectProxy` was :term:`immortal`. This +allowed proxy objects to be directly shared across interpreters, because their +reference count was thread-safe (since it never changed due to immortality). + +This proved to make the implementation significantly more complicated, and +also ended up with a lot of edge cases that would have been a burden on +CPython maintainers. -Immortality seems to be the driver for a lot of complexity in this proposal; -why not use atomic reference counting instead? +Acknowledgements +================ -Atomic reference counting has been tried before in previous :term:`GIL` -removal attempts, but unfortunately added too much overhead to CPython to be -feasible, because atomic "add" operations are much slower than their non-atomic -counterparts. Immortality, while complex, has the benefit of being efficient -and thread-safe without needing to slow down single-threaded performance with -reference counting. +This PEP would not have been possible without discussion and feedback from +Eric Snow, Petr Viktorin, and Yury Selivanov. Copyright ========= From ac75e385752db9e2b34dd2f9e171c223c41995bf Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Wed, 26 Nov 2025 23:33:13 -0500 Subject: [PATCH 4/7] Some rewording. --- peps/pep-0797.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0797.rst b/peps/pep-0797.rst index 3e9f1f1a3f3..c05b08bd577 100644 --- a/peps/pep-0797.rst +++ b/peps/pep-0797.rst @@ -31,14 +31,14 @@ For example:: Background ========== -At the C level, all objects in Python's C API are interacted with through their -type (a pointer to a :c:type:`PyTypeObject`). For example, to call an object, +At the C level, Python interacts with objects through their +type (:c:member:`~PyObject.ob_type`). For example, to call an object, the interpreter will access the :c:member:`~PyTypeObject.tp_call` field on the object's type. This is where the magic of a shared object proxy can happen. The :c:type:`!PyTypeObject` for a shared object proxy must implement wrapping behavior for every single field on the type object structure. -So, going back to ``tp_call``, an object proxy must be able to "intercept" the +So, for ``tp_call``, an object proxy must be able to "intercept" the call in such a way where the wrapped object's ``tp_call`` slot can be executed without thread-safety issues. This is done by switching the :term:`attached thread state`. From 57410a2b708055182483fbd869ef07d5b357723a Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Thu, 27 Nov 2025 18:56:37 -0500 Subject: [PATCH 5/7] Very big revision. --- peps/pep-0797.rst | 334 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 260 insertions(+), 74 deletions(-) diff --git a/peps/pep-0797.rst b/peps/pep-0797.rst index c05b08bd577..e319fae0bd7 100644 --- a/peps/pep-0797.rst +++ b/peps/pep-0797.rst @@ -1,5 +1,5 @@ PEP: 797 -Title: Shared Object Proxies for Subinterpreters +Title: Shared Object Proxies Author: Peter Bierma Discussions-To: Pending Status: Draft @@ -13,56 +13,22 @@ Abstract ======== This PEP introduces a new :func:`~concurrent.interpreters.share` function to -the :mod:`concurrent.interpreters` module, which allows *any* arbitrary object -to be shared across multiple interpreters, at the cost of being less efficient -under multithreaded code while accessing it. +the :mod:`concurrent.interpreters` module, which allows any arbitrary object +to be shared across interpreters using an object proxy, at the cost of being +less efficient under multithreaded code. -For example:: +For example: + +.. code-block:: python from concurrent import interpreters with open("spanish_inquisition.txt") as unshareable: interp = interpreters.create() - proxy = interpreters.share(unshareable) as proxy: + proxy = interpreters.share(unshareable) interp.prepare_main(file=proxy) interp.exec("file.write('I didn't expect the Spanish Inquisition')") - -Background -========== - -At the C level, Python interacts with objects through their -type (:c:member:`~PyObject.ob_type`). For example, to call an object, -the interpreter will access the :c:member:`~PyTypeObject.tp_call` field on the -object's type. This is where the magic of a shared object proxy can happen. - -The :c:type:`!PyTypeObject` for a shared object proxy must implement -wrapping behavior for every single field on the type object structure. -So, for ``tp_call``, an object proxy must be able to "intercept" the -call in such a way where the wrapped object's ``tp_call`` slot can be executed -without thread-safety issues. This is done by switching the -:term:`attached thread state`. - -In the C API, a :term:`thread state` belongs to a certain interpreter, and by -holding an attached thread state, the thread may interact with any object -belonging to its interpreter. This is because holding an attached thread state -implies things like holding the :term:`GIL`, which make object access thread-safe. - -So, with that in mind, the only thing that the object proxy has to do to call -a type slot is hold an attached thread state for the object's interpreter. -This is the fundamental idea of how a shared object proxy works: allow access -from any interpreter, but switch to the wrapped object's interpreter when a type -slot is called. - -Once the attached thread state has been switched to match a wrapped object's -interpreter, arguments and the return value (if it's a ``PyObject *``) of the -slot need to be shared back to the caller. This is done by first attempting to -share them natively (for example, with ``pickle``), and then falling back to -creating a new shared object proxy if all else fails. The new proxy is given -the same context as the current proxy, meaning the newly wrapped object will -be able to be freed once the :func:`~concurrent.interpreters.share` context -is closed. - Motivation ========== @@ -71,16 +37,18 @@ Many Objects Cannot be Shared Between Subinterpreters In Python 3.14, the new :mod:`concurrent.interpreters` module can be used to create multiple interpreters in a single Python process. This works well for -stateless code (that is, code that doesn't need any state from a caller) and -objects that can be serialized, but it is fairly common for applications to -want to use highly-complex data structures (that cannot be serialized) with -their concurrency. - -Currently, :mod:`!concurrent.interpreters` can only share -:ref:`a handful of types ` natively, and then falls back -to the :mod:`pickle` module for other types. This can be very limited, as many -types of objects cannot be pickled. For example, file objects returned by -:func:`open` cannot be serialized through ``pickle``. +code without shared state, but since one of the primary applications of +subinterpreters is to bypass the :term:`global interpreter lock`, it is +fairly common for programs to require highly-complex data structures that are +not easily shareable. In turn, this damages the practicality of +subinterpreters for concurrency. + +As of writing, subinterpreters can only share :ref:`a handful of types +` natively, relying on the :mod:`pickle` module +for other types. This can be very limited, as many types of objects cannot be +serialized with ``pickle`` (such as file objects returned by :func:`open`). +Additionally, serialization can be a very expensive operation, which is not +ideal for multithreaded applications. Rationale ========= @@ -89,12 +57,14 @@ A Fallback for Object Sharing ----------------------------- A shared object proxy is designed to be a fallback for sharing an object -between interpreters, because it does not scale well under multiple threads. -As such, this PEP does not make other mechanisms for sharing objects -(namely, serialization) obsolete. A shared object proxy should only be used as +between interpreters. A shared object proxy should only be used as a last-resort for highly complex objects that cannot be serialized or shared in any other way. +This means that even if this PEP is accepted, there is still benefit in +implementing other methods to share objects between interpreters. + + Specification ============= @@ -104,32 +74,248 @@ Specification This cannot be constructed from Python; instead, use the :func:`~concurrent.interpreters.share` function. - When interacting with the wrapped object, the proxy will switch to the - interpreter in which the object was created. Arguments passed to anything - on the proxy are also wrapped in a new object proxy if the type isn't - natively shareable (so, for example, strings would not be wrapped in an - object proxy, but file objects would). The same goes for return values. - - An object proxy does not scale well under multiple threads; there will - be contention on locks and the :term:`GIL` if it is enabled. - .. function:: concurrent.interpreters.share(obj) Wrap *obj* in a :class:`~concurrent.interpreters.SharedObjectProxy`, - allowing it to be used in other interpreter APIs as if it were natively shareable. + allowing it to be used in other interpreter APIs as if it were natively + shareable. + + If *obj* is natively shareable, this function does not create a proxy and + simply returns *obj*. + + +Interpreter Switching +--------------------- + +When interacting with the wrapped object, the proxy will switch to the +interpreter in which the object was created. This must happen for any access +to the object, such accessing attributes or making modifications to the object's +:term:`reference count`. To visualize, ``foo`` in the following code is only +ever called in the main interpreter, despite being accessed in subinterpreters +through a proxy: + +.. code-block:: python + + from concurrent import interpreters + + def foo(): + assert interpreters.get_current() == interpreters.get_main() + + interp = interpreters.create() + proxy = interpreters.share(foo) + interp.prepare_main(foo=proxy) + interp.exec("foo()") + + +Multithreaded Scaling +--------------------- + +To switch to a wrapped object's interpreter, an object proxy must swap the +:term:`attached thread state` of the current thread, which will in turn wait +on the :term:`GIL` of the target interpreter, if it is enabled. This means that +a shared object proxy will experience contention when accessed concurrently, +but are still useful for multicore threading, since other threads in the +interpreter are free to execute while waiting on the GIL of the target +interpreter. + +As an example, imagine that multiple interpreters want to write a log through +a proxy for the main interpreter, but don't want to constantly wait on the log. +By accessing the proxy in a separate thread for each interpreter, the thread +performing the computation can still execute while accessing the proxy. + +.. code-block:: python + + from concurrent import interpreters + + def write_log(message): + print(message) + + def execute(n, write_log): + from threading import Thread + from queue import Queue + + log = Queue() + + # By performing this in a separate thread, 'execute' can still run + # while the log is being accessed by the main interpreter. + def log_queue_loop(): + while True: + write_log(log.get()) + + thread = Thread(target=log_queue_loop) + thread.start() + + for i in range(100000): + n ** i + log.put(f"Completed an iteration: {i}") + + thread.join() + + proxy = interpreters.share(write_log) + for n in range(4): + interp = interpreters.create() + interp.call_in_thread(execute, n, proxy) + + +Proxy Copying +------------- + +Contrary to what one might think, a shared object proxy itself can only be used +in one interpreter, because the proxy's reference count is not thread-safe +(and thus cannot be accessed from multiple interpreters). Instead, when crossing +an interpreter boundary, a new proxy is created for the target interpreter that +wraps the same object as the original proxy. + +For example, in the following code, there are two proxies created, not just one. + +.. code-block:: python + + from concurrent import interpreters - For example: + interp = interpreters.create() + foo = object() + proxy = interpreters.share(foo) - .. code-block:: python + # The proxy crosses an interpreter boundary here. 'proxy' is *not* directly + # send to 'interp'. Instead, a new proxy is created for 'interp', and the + # reference to 'foo' is merely copied. Thus, both interpreters have their + # own proxy that are wrapping the same object. + interp.prepare_main(proxy=proxy) + +Thread-local State +------------------ + +Accessing an object proxy will retain information stored on the current +:term:`thread state`, such as thread-local variables stored by +:class:`threading.local` and context variables stored by :mod:`contextvars`. +This allows the following case to work correctly: + +.. code-block:: python + + from concurrent import interpreters + from threading import local + + thread_local = local() + thread_local.value = 1 + + def foo(): + assert thread_local.value == 1 + + interp = interpreters.create() + proxy = interpreters.share(foo) + interp.prepare_main(foo=proxy) + interp.exec("foo()") + +In order to retain thread-local data when accessing an object proxy, each +thread will have to keep track of the last used thread state for +each interpreter. In C, this behavior looks like this: + +.. code-block:: c + + // Error checking has been omitted for brevity + PyThreadState *tstate = PyThreadState_New(interp); + + // By swapping the current thread state to 'interp', 'tstate' will be + // associated with 'interp' for the current thread. That means that accessing + // a shared object proxy will use 'tstate' instead of creating its own + // thread state. + PyThreadState *save = PyThreadState_Swap(tstate); + + // 'save' is now the most recently used thread state, so shared object + // proxies in this thread will use it instead of 'tstate' when accessing + // 'interp'. + PyThreadState_Swap(save); + +In the event that no thread state exists for an interpreter in a given thread, +a shared object proxy will create its own thread state that will be owned by +the interpreter (meaning it will not be destroyed until interpreter +finalization), which will persist across all shared object proxy accesses in +the thread. In other words, a shared object proxy ensures that thread local +variables and similar state will not disappear. + + +Memory Management +----------------- + +All proxy objects hold a :term:`strong reference` to the object that they +wrap. As such, destruction of a shared object proxy may trigger destruction +of the wrapped object if the proxy holds the last reference to it, even if +the proxy belongs to a different interpreter. For example: + +.. code-block:: python + + from concurrent import interpreters + + interp = interpreters.create() + foo = object() + proxy = interpreters.share(foo) + interp.prepare_main(proxy=proxy) + del proxy, foo + + # 'foo' is still alive at this point, because the proxy in 'interp' still + # holds a reference to it. Destruction of 'interp' will then trigger the + # destruction of 'proxy', and subsequently the destruction of 'foo'. + interp.close() + + +Shared object proxies support the garbage collector protocol, but will only +traverse the object that they wrap if the garbage collection is occurring +in the wrapped object's interpreter. To visualize: + +.. code-block:: python + + from concurrent import interpreters + import gc + + proxy = interpreters.share(object()) + + # This prints out [], because the object is owned + # by this interpreter. + print(gc.get_referents(proxy)) + + interp = interpreters.create() + interp.prepare_main(proxy=proxy) + + # This prints out [], because the wrapepd object must be invisible to this + # interpreter. + interp.exec("import gc; print(gc.get_referents(proxy))") + + +Interpreter Lifetimes +********************* + +When an interpreter is destroyed, proxies wrapping objects from that +interpreter may still exist elsewhere. To prevent this from causing crashes, +an interpreter will invalidate all proxies pointing its objects by overwriting +their wrapped object with :const:`None`. + +To demonstrate, the following snippet first prints out ``Alive``, and then +``None`` after deleting the interpreter: + +.. code-block:: python + + from concurrent import interpreters + + def test(): from concurrent import interpreters - with open("spanish_inquisition.txt") as unshareable: - interp = interpreters.create() - proxy = interpreters.share(unshareable) - interp.prepare_main(file=proxy) - interp.exec("file.write('I didn't expect the Spanish Inquisition')") + class Test: + def __str__(self): + return "Alive" + + return interpreters.share(Test()) + + interp = interpreters.create() + wrapped = interp.call(test) + print(wrapped) # Alive + interp.close() + print(wrapped) # None + +Note that the proxy is not physically replaced (``wrapped`` in the above example +is still a ``SharedObjectProxy`` instance), but instead has its wrapped object +replaced to ``None``. Backwards Compatibility @@ -146,7 +332,7 @@ How to Teach This ================= New APIs and important information about how to use them will be added to the -:mod:`concurrent.interpreters` documentation. +:mod:`concurrent.interpreters` documentation. Reference Implementation ======================== @@ -173,7 +359,7 @@ Acknowledgements ================ This PEP would not have been possible without discussion and feedback from -Eric Snow, Petr Viktorin, and Yury Selivanov. +Eric Snow, Petr Viktorin, Kirill Podoprigora, and Yury Selivanov. Copyright ========= From 21c6590c1737a825c74165a4e6bb9ca9b65067fa Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Thu, 27 Nov 2025 18:59:48 -0500 Subject: [PATCH 6/7] Fix Sphinx. --- peps/pep-0797.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0797.rst b/peps/pep-0797.rst index e319fae0bd7..5170cf416d1 100644 --- a/peps/pep-0797.rst +++ b/peps/pep-0797.rst @@ -289,7 +289,7 @@ Interpreter Lifetimes When an interpreter is destroyed, proxies wrapping objects from that interpreter may still exist elsewhere. To prevent this from causing crashes, an interpreter will invalidate all proxies pointing its objects by overwriting -their wrapped object with :const:`None`. +their wrapped object with ``None``. To demonstrate, the following snippet first prints out ``Alive``, and then ``None`` after deleting the interpreter: @@ -347,7 +347,7 @@ Directly Sharing Proxy Objects ------------------------------ The initial revision of this proposal took an approach where an instance of -:class:`~conccurent.interpreters.SharedObjectProxy` was :term:`immortal`. This +:class:`~concurrent.interpreters.SharedObjectProxy` was :term:`immortal`. This allowed proxy objects to be directly shared across interpreters, because their reference count was thread-safe (since it never changed due to immortality). From db66d08d6c90bd7e76e0a149d9fac607f5aafc39 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Thu, 27 Nov 2025 19:13:19 -0500 Subject: [PATCH 7/7] Add Adam to acknowledgements. :) --- peps/pep-0797.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0797.rst b/peps/pep-0797.rst index 5170cf416d1..d420b2506c5 100644 --- a/peps/pep-0797.rst +++ b/peps/pep-0797.rst @@ -359,7 +359,7 @@ Acknowledgements ================ This PEP would not have been possible without discussion and feedback from -Eric Snow, Petr Viktorin, Kirill Podoprigora, and Yury Selivanov. +Eric Snow, Petr Viktorin, Kirill Podoprigora, Adam Turner, and Yury Selivanov. Copyright =========