Every second test fails with "RuntimeError: Event loop is closed" when re-using AsyncConnectionPool

[chbndrhnns]

Hi,

I first noticed the issue with httpx and was able to narrow it down to this snippet which exposes the same behaviour but with a shorter exception trace.

I expect four passing tests but every second test is failing.
A naive solution is to change the fixture scope from module to function. However, this impacts me in testing my application which uses httpx deep down under the hood.

import httpcore
import pytest

pytestmark = [pytest.mark.anyio]


@pytest.fixture
def anyio_backend():
    return 'asyncio'


@pytest.fixture(scope="module")
def adapter():
    return httpcore.AsyncConnectionPool()


@pytest.mark.parametrize("count", [1, 2, 3, 4, ])
async def test_(count, adapter):
    assert await adapter.request("GET",
                                 "http://jsonplaceholder.typicode.com/todos/1")

This is the error I see:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/httpcore/_async/interfaces.py:41: in request
    response = await self.handle_async_request(request)
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/httpcore/_async/connection_pool.py:253: in handle_async_request
    raise exc
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/httpcore/_async/connection_pool.py:237: in handle_async_request
    response = await connection.handle_async_request(request)
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/httpcore/_async/connection.py:90: in handle_async_request
    return await self._connection.handle_async_request(request)
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/httpcore/_async/http11.py:104: in handle_async_request
    await self._response_closed()
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/httpcore/_async/http11.py:209: in _response_closed
    await self.aclose()
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/httpcore/_async/http11.py:217: in aclose
    await self._network_stream.aclose()
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/httpcore/backends/asyncio.py:52: in aclose
    await self._stream.aclose()
../../Library/Caches/pypoetry/virtualenvs/pythonproject3-nIT-Z9on-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py:1323: in aclose
    self._transport.close()
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/selector_events.py:700: in close
    self._loop.call_soon(self._call_connection_lost, None)
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py:746: in call_soon
    self._check_closed()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_UnixSelectorEventLoop running=False closed=True debug=False>

    def _check_closed(self):
        if self._closed:
>           raise RuntimeError('Event loop is closed')
E           RuntimeError: Event loop is closed

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py:510: RuntimeError

[zanieb]

Hey @chbndrhnns — I believe your problem is occurring because the event loop is scoped per function but your pool exists in a broader scope and then cannot access the event loop its connections were created with to clean up resources.

If you close the connection pool after each test run, all your tests pass:

@pytest.mark.parametrize(
    "count",
    [
        1,
        2,
        3,
        4,
    ],
)
async def test_(count, adapter):
    assert await adapter.request("GET", "http://jsonplaceholder.typicode.com/todos/1")
    await adapter.aclose()

You could do this in a separate fixture:

@pytest.fixture(autouse=True)
async def close_adapter(adapter):
    yield
    await adapter.aclose()

But at this point, you might as well context manage the pool correctly. Attempting to do this clarifies the issue:

@pytest.fixture(scope="module")
async def adapter():
    async with httpcore.AsyncConnectionPool() as pool:
        yield pool

ScopeMismatch: You tried to access the function scoped fixture event_loop with a module scoped request object, involved factories:
example.py:28:  def adapter()

To resolve that, you can define an event loop that is not function scoped:

@pytest.fixture(scope="session")
def event_loop(request):
    """
    Redefine the event loop to support session/module-scoped fixtures;
    see https://github.com/pytest-dev/pytest-asyncio/issues/68
    """
    policy = asyncio.get_event_loop_policy()
    loop = policy.new_event_loop()

    try:
        yield loop
    finally:
        loop.close()

Now all of your tests pass.

[chbndrhnns]

Thanks for the details here! I think what really confused me is that it's exactly every second test fails in the case where I manage the loop or the lifecycle not correctly. Do you have any idea why that is?

Also, this approach also fixes my tests: Using a larger scope for anyio_backend and making the adapter fixture async.

@pytest.fixture(scope="session")
def anyio_backend():
    return "asyncio"


@pytest.fixture(scope="module")
async def adapter():
    return httpcore.AsyncConnectionPool()

[zanieb]

Thanks for the details here! I think what really confused me is that it's exactly every second test fails in the case where I manage the loop or the lifecycle not correctly. Do you have any idea why that is?

Not precisely, but I'd guess each second test probably attempts to perform some cleanup and fails since the loop is not present from the previous test but still drops the connection from the pool so the next test succeeds.

[ewjoachim]

Interestingly, I believe this is the root issue of pydantic/pydantic-ai#748, which uses httpx > httpcore.

pydantic-ai reuses httpx transports accross calls (which reuses AsyncConnectionPool), even when those calls might happen in different event loops. Would you say the issue comes from pydantic-ai that does that, or should httpcore refuse to reuse a Connection in a pool that was created as part of a different loop ?

[ewjoachim]

Ha, so there's an issue downstream but also upstream: agronholm/anyio#743

[lovelydinosaur]

should httpcore refuse to reuse a Connection in a pool that was created as part of a different loop ?

As @zanieb mentions... "you might as well context manage the pool correctly".

We could be helping users out by strictly requiring content-managed clients. That'd ensure proper resource scoping.

We don't happen to do that currently due to historical reasons.

---

Eg. aiohttp sessions enforce this style.

[ewjoachim]

Thanks for weighing in!