Skip to content

The create_ssl_context() call can consume a lot of memory #3734

@nascheme

Description

@nascheme

Based on researching this bug in Python, the create_ssl_context() calls can consume a lot of memory (10s or 100s of megabytes). This problem is made worse by the reference cycles created by BoundSyncStream and BoundAsyncStream (I expect they are keeping the SSL context alive).

It would seem prudent to investigate if the default context can be shared between connections (e.g. use a cache?). The script that shows high memory usage is pretty trivial:

# leak_ssl_local.py
#
# /// script
# requires-python = ">=3.13"
# dependencies = [
#   "httpx",
# ]
# ///
import httpx
import asyncio
import gc
import os
import sys


def get_rss():
    with open(f"/proc/{os.getpid()}/status") as fp:
        for line in fp:
            if line.startswith("VmRSS"):
                rss = line.split(":", 1)[1].rstrip()
                rss = rss.removesuffix(" kB")
                return int(rss)
    return None


# Can run: python3 -m http.server 8001
URL = 'http://localhost:8001/'


async def _main() -> None:
    for i in range(3000):
        async with httpx.AsyncClient() as client:
            resp = await client.get(URL)
            await resp.aclose()
            await client.aclose()
            if True:
                # this breaks the reference cycle
                resp.stream = None

        rss = get_rss()
        print(f"Loop #{i}: {rss:,} kB")


if __name__ == "__main__":
    asyncio.run(_main())

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions