Skip to content

PYTHON-5044 - Fix successive AsyncMongoClients on a single loop always ti… #2065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jan 22, 2025
Merged

PYTHON-5044 - Fix successive AsyncMongoClients on a single loop always ti… #2065

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5a1e7cb
PYTHON-5044 - Successive AsyncMongoClients on a single loop always ti…
NoahStapp Jan 17, 2025
cdc2cf0
Only join executors on async
NoahStapp Jan 21, 2025
b898826
Remove unneeded reset_async_client_context
NoahStapp Jan 21, 2025
6547c3c
async spec runner should only call kill_all_sessions after runs
NoahStapp Jan 21, 2025
f1f8d62
Revert all cleanup changes
NoahStapp Jan 21, 2025
a4eedb3
Remove task cancelling from AsyncPeriodicExecutor.close
NoahStapp Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions pymongo/asynchronous/mongo_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1565,6 +1565,8 @@ async def close(self) -> None:
# TODO: PYTHON-1921 Encrypted MongoClients cannot be re-opened.
await self._encrypter.close()
self._closed = True
# Yield to the asyncio event loop so all executor tasks properly exit after cancellation
await asyncio.sleep(0)
Copy link
Member

@ShaneHarvey ShaneHarvey Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yielding here doesn't guarantee anything about task cleanup. Shouldn't we actually await the tasks to ensure they are cleaned up properly?


if not _IS_SYNC:
# Add support for contextlib.aclosing.
Expand Down
31 changes: 19 additions & 12 deletions pymongo/network_layer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,18 +267,25 @@ async def async_receive_data(
else:
read_task = create_task(_async_receive(sock, length, loop)) # type: ignore[arg-type]
tasks = [read_task, cancellation_task]
done, pending = await asyncio.wait(
tasks, timeout=timeout, return_when=asyncio.FIRST_COMPLETED
)
for task in pending:
task.cancel()
if pending:
await asyncio.wait(pending)
if len(done) == 0:
raise socket.timeout("timed out")
if read_task in done:
return read_task.result()
raise _OperationCancelled("operation cancelled")
try:
done, pending = await asyncio.wait(
tasks, timeout=timeout, return_when=asyncio.FIRST_COMPLETED
)
for task in pending:
task.cancel()
if pending:
await asyncio.wait(pending)
if len(done) == 0:
raise socket.timeout("timed out")
if read_task in done:
return read_task.result()
raise _OperationCancelled("operation cancelled")
except asyncio.CancelledError:
for task in tasks:
task.cancel()
await asyncio.wait(tasks)
raise

finally:
sock.settimeout(sock_timeout)

Expand Down
2 changes: 2 additions & 0 deletions pymongo/periodic_executor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ def close(self, dummy: Any = None) -> None:
callback; see monitor.py.
"""
self._stopped = True
if self._task:
self._task.cancel()
Copy link
Member

@ShaneHarvey ShaneHarvey Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering how this relates to the issue described in DRIVERS-3076. Like will calling cancel here change the user visible events a Monitor emits on close()?

Copy link
Member

@ShaneHarvey ShaneHarvey Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping.

Copy link
Contributor Author

@NoahStapp NoahStapp Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a task is cancelled, it should stop executing on the next iteration of the event loop. Since I believe the CancelledError is thrown from the next await call inside the cancelled task, it's possible that the Monitor emits events differently between cancellations.

Copy link
Member

@ShaneHarvey ShaneHarvey Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, if we use this approach then we won't emit the expected ServerHeartbeatFailedEvent on cancellation. Do we need this change in this PR anymore? Can we defer it?

Copy link
Contributor Author

@NoahStapp NoahStapp Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Now that we aren't awaiting background tasks on close in this PR, this change is unneeded.


async def join(self, timeout: Optional[int] = None) -> None:
if self._task is not None:
Expand Down
1 change: 1 addition & 0 deletions pymongo/synchronous/mongo_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1559,6 +1559,7 @@ def close(self) -> None:
# TODO: PYTHON-1921 Encrypted MongoClients cannot be re-opened.
self._encrypter.close()
self._closed = True
# Yield to the asyncio event loop so all executor tasks properly exit after cancellation

if not _IS_SYNC:
# Add support for contextlib.closing.
Expand Down
Loading