Skip to content

Use standard libary Zstandard for Python 3.14+ #3613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mollymorphous wants to merge 2 commits into from
+29 −16
Closed

Use standard libary Zstandard for Python 3.14+ #3613

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b312e20
feat: Use standard library zstd (3.14+) if available
mollymorphous Jul 25, 2025
824f3f5
docs: Default zstd support for Python 3.14+
mollymorphous Jul 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ As well as these optional installs:
* `rich` - Rich terminal support. *(Optional, with `httpx[cli]`)*
* `click` - Command line client support. *(Optional, with `httpx[cli]`)*
* `brotli` or `brotlicffi` - Decoding for "brotli" compressed responses. *(Optional, with `httpx[brotli]`)*
* `zstandard` - Decoding for "zstd" compressed responses. *(Optional, with `httpx[zstd]`)*
* `zstandard` - Decoding for "zstd" compressed responses before Python 3.14. *(Optional, with `httpx[zstd]`)*

A huge amount of credit is due to `requests` for the API layout that
much of this work follows, as well as to `urllib3` for plenty of design
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ As well as these optional installs:
* `rich` - Rich terminal support. *(Optional, with `httpx[cli]`)*
* `click` - Command line client support. *(Optional, with `httpx[cli]`)*
* `brotli` or `brotlicffi` - Decoding for "brotli" compressed responses. *(Optional, with `httpx[brotli]`)*
* `zstandard` - Decoding for "zstd" compressed responses. *(Optional, with `httpx[zstd]`)*
* `zstandard` - Decoding for "zstd" compressed responses before Python 3.14. *(Optional, with `httpx[zstd]`)*

A huge amount of credit is due to `requests` for the API layout that
much of this work follows, as well as to `urllib3` for plenty of design
Expand Down
5 changes: 3 additions & 2 deletions docs/quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,9 @@ b'<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>...'

Any `gzip` and `deflate` HTTP response encodings will automatically
be decoded for you. If `brotlipy` is installed, then the `brotli` response
encoding will be supported. If `zstandard` is installed, then `zstd`
response encodings will also be supported.
encoding will be supported. The `zstd` response encoding is supported by
default on Python 3.14 and later, and optionally available on earlier Python
versions with `zstandard` installed.

For example, to create an image from binary data returned by a request, you can use the following code:

Expand Down
28 changes: 18 additions & 10 deletions httpx/_decoders.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,15 @@
brotli = None


# Zstandard support is optional
# Zstandard support is avaible in the standard library from 3.14 and later,
# or by an optional dependency on `zstandard`
try:
import zstandard
import compression.zstd as zstandard
except ImportError: # pragma: no cover
zstandard = None # type: ignore
try:
import zstandard
except ImportError: # pragma: no cover
zstandard = None


class ContentDecoder:
Expand Down Expand Up @@ -174,9 +178,16 @@ def __init__(self) -> None:
"Make sure to install httpx using `pip install httpx[zstd]`."
) from None

self.decompressor = zstandard.ZstdDecompressor().decompressobj()
self._new_decompressor()
self.seen_data = False

def _new_decompressor(self) -> None:
decompressor = zstandard.ZstdDecompressor()
if hasattr(decompressor, "decompressobj"):
self.decompressor = decompressor.decompressobj() # prgama: no cover
else:
self.decompressor = decompressor # pragma: no cover
Comment on lines +184 to +189
Copy link
Contributor

@lovelydinosaur lovelydinosaur Jul 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain this part?

Copy link
Author

@mollymorphous mollymorphous Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is admittedly a little awkward. Python upstreamed the pyzstd package into compress.zstd because it's API was closer to existing standard library compression APIs. The zstandard package provides a ZstdDecompressObj facade that implements the standard library style API.

An alternative would be to import the libraries under separate names so the method would look more like this:

def _new_compressor(self) -> None:
    if compression_zstd is not None:
        self.decompressor = compression_zstd.ZstdDecompressor()
    else:
        self.decompressor = zstandard.ZstdDecompressor().decompressobj()

What do you think?

Copy link

@ddelange ddelange Oct 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the requirement in pyproject.toml is changed to the 3.9+ backport at backports.zstd published by the pyzstd maintainer (which got moved into cpython 3.14 standard library):

backports-zstd==1.0.0 ; python_version < "3.14"

this diff would get much simpler right?

maybe with a simpler diff, this PR has a better chance of landing?

reading from chunked streams gets streamlined like that: aio-libs/aiohttp@df8ad83


def decode(self, data: bytes) -> bytes:
assert zstandard is not None
self.seen_data = True
Expand All @@ -185,19 +196,16 @@ def decode(self, data: bytes) -> bytes:
output.write(self.decompressor.decompress(data))
while self.decompressor.eof and self.decompressor.unused_data:
unused_data = self.decompressor.unused_data
self.decompressor = zstandard.ZstdDecompressor().decompressobj()
self._new_decompressor()
output.write(self.decompressor.decompress(unused_data))
except zstandard.ZstdError as exc:
raise DecodingError(str(exc)) from exc
return output.getvalue()

def flush(self) -> bytes:
if not self.seen_data:
return b""
ret = self.decompressor.flush() # note: this is a no-op
if not self.decompressor.eof:
if self.seen_data and not self.decompressor.eof:
raise DecodingError("Zstandard data is incomplete") # pragma: no cover
return bytes(ret)
return b""


class MultiDecoder(ContentDecoder):
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ socks = [
"socksio==1.*",
]
zstd = [
"zstandard>=0.18.0",
"zstandard>=0.18.0; python_version < \"3.14\"",
]

[project.scripts]
Expand Down
6 changes: 5 additions & 1 deletion tests/test_decoders.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,11 @@

import chardet
import pytest
import zstandard as zstd

try:
from compression import zstd
except ImportError:
import zstandard as zstd

import httpx

Expand Down