Skip to content

gh-85222: Document the side effect in multiprocessing #136426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aisk wants to merge 11 commits into
base: main
Choose a base branch
from
Open

gh-85222: Document the side effect in multiprocessing #136426

Changes from 4 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
646225c
Document the ctx parameter in some types in multiprocessing
aisk Jul 8, 2025
1b7a0ea
Update side effect document
aisk Jul 8, 2025
29b5cf3
Update document
aisk Jul 8, 2025
7962657
Fix marker
aisk Jul 8, 2025
ed60127
argument -> parameters
aisk Jul 10, 2025
881a347
Reduce duplication
aisk Jul 10, 2025
de61926
calling -> instantiating
aisk Jul 11, 2025
982c7c2
Update Doc/library/multiprocessing.rst
aisk Jul 12, 2025
c85562f
Apply suggestions from code review
aisk Jul 12, 2025
8a9ae53
Update Doc/library/multiprocessing.rst
aisk Jul 12, 2025
a62a4fe
Fix line-break
aisk Jul 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 66 additions & 8 deletions Doc/library/multiprocessing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -867,6 +867,10 @@
locks/semaphores. When a process first puts an item on the queue a feeder
thread is started which transfers objects from a buffer into the pipe.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.
Copy link
Contributor

@LamentXU123 LamentXU123 Jul 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think repeating the same words here may be a bit 'noisy'. Could we just mention it once?

Copy link
Contributor Author

@aisk aisk Jul 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern is that if a user only uses one of these functions and checks that function's documentation, and unfortunately that is not the function that contains this document, they may mess it up.

But I'm unsure about this.

Copy link
Contributor

@sharktide sharktide Jul 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be biased but I don't think the extra clarity hurts... I'm with @aisk

Copy link
Member

@gpshead gpshead Jul 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I asked for this to be mentioned everywhere for that reason as this is non-obvious behavior of multiprocessing. This could probably be refined a bit, i'll point some docs focused reviewers at it.

This comment was marked as off-topic.

Sign in to view

Copy link
Contributor

@LamentXU123 LamentXU123 Jul 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern is that if a user only uses one of these functions and checks that function's documentation, and unfortunately that is not the function that contains this document, they may mess it up.

But I'm unsure about this.

IMO it may be better if we mentioned it once and create a reference link to the mention in every function. I'm not sure too (｡･ω･｡)

Copy link
Member

@encukou encukou Jul 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The repeated part could be a single sentence like Ensures that the current global start method is set., with ”global start method” linking to a longer explanation in a dedicated section.

Copy link
Member

@nedbat nedbat Jul 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it needs to be mentioned in every method. There's a section at the top about contexts that can say the global start method is set by any function that does real work (or some appropriate language). I find we often have overarching concerns that apply throughout a module and don't repeat them. Sometime people have to read widely in a module page in order to understand all the nuances.

Maybe I missed the discussion: why is this particular caveat important enough to sprinkle everywhere? What footgun are we preventing?

Copy link
Member

@gpshead gpshead Jul 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The callout about the surprising behavior is also (in addition to the issue for this PR) due to #109070 which only just added the caveat about .get_context() having the setting side effect behavior via https://github.com/python/cpython/pull/136341/files.

I agree that the https://docs.python.org/3.15/library/multiprocessing.html#contexts-and-start-methods section up top should be more clear about this. It really only ever mentions get_context() as an alternative today and never highlights that the context is implicitly set at instantiation time by all sorts of APIs... We do say To select a start method you use the set_start_method() in the if __name__ == '__main__' clause of the main module. but we never actually explain the restrictions leading to that advice, the why, there... That would help.

After that these mentions about context setting could turn into a single sentence similar to what encoku suggested that ref's back to that section.

(the other part of this PR is documenting which things accept a ctx= parameter at all)

Copy link
Contributor Author

@aisk aisk Jul 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thank you all for review this PR, I updated the document and added a new section to explain the implicitly set of start method, and changed other methods/types to link to this section. Please review it to see if this is suitable.


The usual :exc:`queue.Empty` and :exc:`queue.Full` exceptions from the
standard library's :mod:`queue` module are raised to signal timeouts.

Expand Down Expand Up @@ -977,6 +981,10 @@

It is a simplified :class:`Queue` type, very close to a locked :class:`Pipe`.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

.. method:: close()

Close the queue: release internal resources.
Expand Down Expand Up @@ -1007,6 +1015,10 @@
:class:`JoinableQueue`, a :class:`Queue` subclass, is a queue which
additionally has :meth:`task_done` and :meth:`join` methods.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

.. method:: task_done()

Indicate that a formerly enqueued task is complete. Used by queue
Expand Down Expand Up @@ -1361,13 +1373,21 @@

A barrier object: a clone of :class:`threading.Barrier`.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

.. versionadded:: 3.3

.. class:: BoundedSemaphore([value])

A bounded semaphore object: a close analog of
:class:`threading.BoundedSemaphore`.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

A solitary difference from its close analog exists: its ``acquire`` method's
first argument is named *block*, as is consistent with :meth:`Lock.acquire`.

Expand All @@ -1388,13 +1408,20 @@
If *lock* is specified then it should be a :class:`Lock` or :class:`RLock`
object from :mod:`multiprocessing`.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

.. versionchanged:: 3.3
The :meth:`~threading.Condition.wait_for` method was added.

.. class:: Event()

A clone of :class:`threading.Event`.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

.. class:: Lock()

Expand All @@ -1410,6 +1437,10 @@
instance of ``multiprocessing.synchronize.Lock`` initialized with a
default context.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

:class:`Lock` supports the :term:`context manager` protocol and thus may be
used in :keyword:`with` statements.

Expand Down Expand Up @@ -1467,6 +1498,10 @@
instance of ``multiprocessing.synchronize.RLock`` initialized with a
default context.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

:class:`RLock` supports the :term:`context manager` protocol and thus may be
used in :keyword:`with` statements.

Expand Down Expand Up @@ -1526,6 +1561,10 @@

A semaphore object: a close analog of :class:`threading.Semaphore`.

If the global start method has not been set, calling this function will
have the side effect of setting the current global start method.
See the :func:`get_context` function.

A solitary difference from its close analog exists: its ``acquire`` method's
first argument is named *block*, as is consistent with :meth:`Lock.acquire`.

Expand Down Expand Up @@ -1660,7 +1699,7 @@
attributes which allow one to use it to store and retrieve strings -- see
documentation for :mod:`ctypes`.

.. function:: Array(typecode_or_type, size_or_initializer, *, lock=True)
.. function:: Array(typecode_or_type, size_or_initializer, *, lock=True, ctx=None)

The same as :func:`RawArray` except that depending on the value of *lock* a
process-safe synchronization wrapper may be returned instead of a raw ctypes
Expand All @@ -1674,9 +1713,14 @@
automatically protected by a lock, so it will not necessarily be
"process-safe".

Note that *lock* is a keyword-only argument.
*ctx* is a context object, or ``None`` (use the current context). If ``None``,

Check warning on line 1716 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs 

py:func reference target not found: get_context [ref.func]
calling this function will have the side effect of setting the current global
start method if it has not been set already. See the :func:`get_context`
function.

.. function:: Value(typecode_or_type, *args, lock=True)
Note that *lock* and *ctx* are keyword-only argument.

.. function:: Value(typecode_or_type, *args, lock=True, ctx=None)

The same as :func:`RawValue` except that depending on the value of *lock* a
process-safe synchronization wrapper may be returned instead of a raw ctypes
Expand All @@ -1689,20 +1733,30 @@
automatically protected by a lock, so it will not necessarily be
"process-safe".

Note that *lock* is a keyword-only argument.
*ctx* is a context object, or ``None`` (use the current context). If ``None``,

Check warning on line 1736 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs 

py:func reference target not found: get_context [ref.func]
calling this function will have the side effect of setting the current global
start method if it has not been set already. See the :func:`get_context`
function.

Note that *lock* and *ctx* are keyword-only argument.

.. function:: copy(obj)

Return a ctypes object allocated from shared memory which is a copy of the
ctypes object *obj*.

.. function:: synchronized(obj[, lock])
.. function:: synchronized(obj, lock=None, ctx=None)

Return a process-safe wrapper object for a ctypes object which uses *lock* to
synchronize access. If *lock* is ``None`` (the default) then a
:class:`multiprocessing.RLock` object is created automatically.

*ctx* is a context object, or ``None`` (use the current context). If ``None``,

Check warning on line 1754 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs 

py:func reference target not found: get_context [ref.func]
calling this function will have the side effect of setting the current global
start method if it has not been set already. See the :func:`get_context`
function.

A synchronized wrapper will have two methods in addition to those of the

Check warning on line 1759 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs 

py:meth reference target not found: get_lock [ref.meth]

Check warning on line 1759 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs 

py:meth reference target not found: get_obj [ref.meth]
object it wraps: :meth:`get_obj` returns the wrapped object and
:meth:`get_lock` returns the lock object used for synchronization.

Expand Down Expand Up @@ -1819,8 +1873,10 @@
*serializer* must be ``'pickle'`` (use :mod:`pickle` serialization) or
``'xmlrpclib'`` (use :mod:`xmlrpc.client` serialization).

*ctx* is a context object, or ``None`` (use the current context). See the
:func:`get_context` function.
*ctx* is a context object, or ``None`` (use the current context). If ``None``,

Check warning on line 1876 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs 

py:func reference target not found: get_context [ref.func]
calling this function will have the side effect of setting the current global
start method if it has not been set already. See the :func:`get_context`
function.

*shutdown_timeout* is a timeout in seconds used to wait until the process
used by the manager completes in the :meth:`shutdown` method. If the
Expand Down Expand Up @@ -2309,11 +2365,13 @@
unused resources to be freed. The default *maxtasksperchild* is ``None``, which
means worker processes will live as long as the pool.

*context* can be used to specify the context used for starting

Check warning on line 2368 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs 

py:func reference target not found: get_context [ref.func]

Check warning on line 2368 in Doc/library/multiprocessing.rst

View workflow job for this annotation

GitHub Actions / Docs / Docs 

py:func reference target not found: multiprocessing.Pool [ref.func]
the worker processes. Usually a pool is created using the
function :func:`multiprocessing.Pool` or the :meth:`Pool` method
of a context object. In both cases *context* is set
appropriately.
appropriately. If ``None``, calling this function will have the side effect
of setting the current global start method if it has not been set already.
Copy link
Contributor

@sharktide sharktide Jul 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
of setting the current global start method if it has not been set already.
of setting the current global start method to the
system default if it has not been set already.

More small suggestions to improve reading flow

Up to you if you wanna take this one

See the :func:`get_context` function.

Note that the methods of the pool object should only be called by
the process which created the pool.
Expand Down
Loading