-
-
Notifications
You must be signed in to change notification settings - Fork 33.6k
GH-100108: Add async generators best practices section #141885
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 3 commits
8fb10ee
ae50f3f
99f148c
11242b4
3556aef
a70038a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -248,3 +248,104 @@ Output in debug mode:: | |
| File "../t.py", line 4, in bug | ||
| raise Exception("not consumed") | ||
| Exception: not consumed | ||
|
|
||
|
|
||
| Asynchronous generators best practices | ||
| ====================================== | ||
|
|
||
| By :term:`asynchronous generator` in this section we will mean | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems obvious, by generator we typically mean the generator object not the function.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I saw this clarification in few places (e.g. https://docs.python.org/3/glossary.html#term-asynchronous-generator), so I thought it was worth mentioning. I'm OK to remove this if you think so.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In my experience, the term "generator" is used equally for a generator function and for the iterator returned by calling a generator function. If you really only use it for the latter, then this clarification makes sense. |
||
| an :term:`asynchronous generator iterator` that returned by | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| :term:`asynchronous generator` function. | ||
|
|
||
| Manually close the generator | ||
| ---------------------------- | ||
|
|
||
| If an asynchronous generator happens to exit early by :keyword:`break`, the caller | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This feels too heavy for start, I would start it as "It is recommended to manually close the generator... " then in second para describe the issue with breaking early.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, will fix this.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IIUC the scenario(s) you are trying to describe is where the code iterating over the generator doesn't iterate till the end. If you're iterating over it using a for-loop (there are other ways!) that could be a break in that for-loop, or the task containing that for-loop getting canceled, or indeed an exception being raised in the for-loop body. But I'm not sure it's helpful to try to list all the reasons why this can happen -- maybe the most useful one to mention is an exception being raised during that for-loop? (The example code clarifies a lot!) |
||
| task being cancelled, or other exceptions, the generator's async cleanup code | ||
| will run and possibly raise exceptions or access context variables in an | ||
| unexpected context--perhaps after the lifetime of tasks it depends, or | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| during the event loop shutdown when the async-generator garbage collection hook | ||
| is called. | ||
|
|
||
| To prevent this, it is recommended to explicitly close the async generator by | ||
| calling :meth:`~agen.aclose` method, or using :func:`contextlib.aclosing` context | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| manager:: | ||
|
|
||
| import asyncio | ||
| import contextlib | ||
|
|
||
| async def gen(): | ||
| yield 1 | ||
| yield 2 | ||
|
|
||
| async def func(): | ||
| async with contextlib.aclosing(gen()) as g: | ||
| async for x in g: | ||
| break | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| asyncio.run(func()) | ||
|
|
||
| Only create a generator when a loop is already running | ||
| ------------------------------------------------------ | ||
|
|
||
| As said above, if an asynchronous generator is not resumed before it is | ||
| finalized, then any finalization procedures will be delayed. The event loop | ||
| handles this situation and doing it best to call async generator-iterator's | ||
| :meth:`~agen.aclose` at the proper moment, thus allowing any pending | ||
| :keyword:`!finally` clauses to execute. | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Then it is recomended to create async generators only after the event loop | ||
| has already been created. | ||
|
||
|
|
||
| Avoid iterating and closing the same generator concurrently | ||
| ----------------------------------------------------------- | ||
|
|
||
| The async generators allow to be reentered while another | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| :meth:`~agen.aclose`/:meth:`~agen.aclose`/:meth:`~agen.aclose` call is in | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| progress. This may lead to an inconsistent state of the async generator | ||
| and cause errors. | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Let's consider following example:: | ||
|
|
||
| import asyncio | ||
|
|
||
| async def consumer(): | ||
| for idx in range(100): | ||
| await asyncio.sleep(0) | ||
| message = yield idx | ||
| print('received', message) | ||
|
|
||
| async def amain(): | ||
| agenerator = consumer() | ||
| await agenerator.asend(None) | ||
|
|
||
| fa = asyncio.create_task(agenerator.asend('A')) | ||
| fb = asyncio.create_task(agenerator.asend('B')) | ||
| await fa | ||
| await fb | ||
|
|
||
| asyncio.run(amain()) | ||
|
|
||
| Output:: | ||
|
|
||
| received A | ||
| Traceback (most recent call last): | ||
| File "test.py", line 38, in <module> | ||
| asyncio.run(amain()) | ||
| ~~~~~~~~~~~^^^^^^^^^ | ||
| File "Lib\asyncio\runners.py", line 204, in run | ||
| return runner.run(main) | ||
| ~~~~~~~~~~^^^^^^ | ||
| File "Lib\asyncio\runners.py", line 127, in run | ||
| return self._loop.run_until_complete(task) | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ | ||
| File "Lib\asyncio\base_events.py", line 719, in run_until_complete | ||
| return future.result() | ||
| ~~~~~~~~~~~~~^^ | ||
| File "test.py", line 36, in amain | ||
| await fb | ||
| RuntimeError: anext(): asynchronous generator is already running | ||
|
|
||
|
|
||
| Therefore, it is recommended to avoid using the async generators in parallel | ||
| tasks or in the multiple event loops. | ||
sergey-miryanov marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd insert some friendly words here introducing the reader to the purpose of this section. Right now it looks rather stern.