Skip to content

Commit 6d95984

Browse files
aidandjAidan Jensen
aidandj
and
Aidan Jensen
authored
Protobuf 6.32+ update (#690)
* Drop support for py_generic_services * Drop testing support for <6.32 Signed-off-by: Aidan Jensen <aidan.jensen@robust.ai> Co-authored-by: Aidan Jensen <aidan.jensen@robust.ai>
1 parent 325a62b commit 6d95984

24 files changed

+302
-143
lines changed

‎.github/workflows/main.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ jobs:
3030
- 3.13.9
3131
- 3.14.0
3232
protobuf-version:
33-
- 5.28.3
33+
- 6.32.1
3434
- 6.33.1
3535

3636
steps:
@@ -96,7 +96,7 @@ jobs:
9696
- uses: actions/checkout@v4
9797
- uses: actions/setup-python@v5
9898
with:
99-
python-version: "3.8"
99+
python-version: "3.9"
100100
- name: Run formatters and linters
101101
run: |
102102
pip3 install black==24.3.0 isort flake8 flake8-pyi flake8-noqa flake8-bugbear
@@ -116,7 +116,7 @@ jobs:
116116
- uses: actions/checkout@v4
117117
- uses: actions/setup-python@v5
118118
with:
119-
python-version: "3.8"
119+
python-version: "3.9"
120120
- name: Read versions
121121
run: echo ::set-output name=PROTOBUF_VERSION::$(grep "^protobuf>=" test_requirements.txt | cut -f2 -d=)
122122
id: read_versions

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,11 @@
1+
## Upcoming
2+
3+
- Drop support for `py_generic_services` as it was removed from the protobuf compiler starting in version 6.30
4+
- https://protobuf.dev/news/2024-10-02/#rpc-service-interfaces
5+
- Drop testing support for protobuf <6.32 because they don't support editions
6+
- With some more work this could be added back in a testing refactor
7+
- Protobuf <6.32 still had the edition enums and field options, so it *should* still work. But is untested
8+
19
## 3.7.0
210

311
- Mark top-level mangled identifiers as `TypeAlias`.

‎README.md‎

Lines changed: 119 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,24 +15,24 @@ See [Changelog](CHANGELOG.md) for recent changes.
1515

1616
Earlier releases might work, but aren't tested
1717

18-
- [protoc >= 23.4](https://github.com/protocolbuffers/protobuf/releases)
19-
- [python-protobuf >= 5.28.2](https://pypi.org/project/protobuf/) - matching protoc release
20-
- [python >= 3.8](https://www.python.org/downloads/source/) - for running mypy-protobuf plugin.
18+
- [protoc >= 32.0](https://github.com/protocolbuffers/protobuf/releases)
19+
- [python-protobuf >= 6.32](https://pypi.org/project/protobuf/) - matching protoc release
20+
- [python >= 3.9](https://www.python.org/downloads/source/) - for running mypy-protobuf plugin.
2121

2222
## Requirements to run typecheckers on stubs generated by mypy-protobuf
2323

2424
Earlier releases might work, but aren't tested
2525

26-
- [mypy >= v1.11.2](https://pypi.org/project/mypy) or [pyright >= 1.1.383](https://github.com/microsoft/pyright)
27-
- [python-protobuf >= 5.28.2](https://pypi.org/project/protobuf/) - matching protoc release
28-
- [types-protobuf >= 5.28](https://pypi.org/project/types-protobuf/) - for stubs from the google.protobuf library
26+
- [mypy >= v1.14.0](https://pypi.org/project/mypy) or [pyright >= 1.1.383](https://github.com/microsoft/pyright)
27+
- [python-protobuf >= 6.32](https://pypi.org/project/protobuf/) - matching protoc release
28+
- [types-protobuf >= 6.32](https://pypi.org/project/types-protobuf/) - for stubs from the google.protobuf library
2929

3030
### To run typecheckers on code generated with grpc plugin - you'll additionally need
3131

3232
Earlier releases might work, but aren't tested
3333

34-
- [grpcio>=1.66.2](https://pypi.org/project/grpcio/)
35-
- [grpcio-tools>=1.66.2](https://pypi.org/project/grpcio-tools/)
34+
- [grpcio>=1.70](https://pypi.org/project/grpcio/)
35+
- [grpcio-tools>=1.70](https://pypi.org/project/grpcio-tools/)
3636
- [grpc-stubs>=1.53.0.5](https://pypi.org/project/grpc-stubs/)
3737

3838
Other configurations may work, but are not continuously tested currently.
@@ -92,6 +92,117 @@ an executable protoc-gen-mypy. On windows it installs to `protoc-gen-mypy.exe`
9292

9393
See [Changelog](CHANGELOG.md) for full listing
9494

95+
### Differences between the protobuf compiler (`--pyi_out`) and `mypy-protobuf`
96+
97+
(As of 11/17/2025)
98+
99+
* `mypy-protobuf` generates stubs for GRPC services and clients
100+
* `mypy-protobuf` generates correctly typed `HasField` methods depending on field presence, `pyi_out` does not type `HasField` arguments
101+
* `mypy-protobuf` generates correctly typed constructors dependinding on field presence.
102+
* `mypy-protobuf` generates correctly typed `HasField`, `WhichOneof`, and `ClearField` methods.
103+
* There are differences in how `mypy-protobuf` and `pyi_out` generate enums. See [this issue](https://github.com/protocolbuffers/protobuf/issues/8175) for details
104+
105+
#### Examples
106+
107+
`mypy-protobuf`:
108+
109+
```python
110+
import builtins
111+
import google.protobuf.descriptor
112+
import google.protobuf.message
113+
import sys
114+
import typing
115+
116+
if sys.version_info >= (3, 10):
117+
import typing as typing_extensions
118+
else:
119+
import typing_extensions
120+
121+
DESCRIPTOR: google.protobuf.descriptor.FileDescriptor
122+
123+
@typing.final
124+
class Editions2024SubMessage(google.protobuf.message.Message):
125+
DESCRIPTOR: google.protobuf.descriptor.Descriptor
126+
127+
THING_FIELD_NUMBER: builtins.int
128+
thing: builtins.str
129+
def __init__(
130+
self,
131+
*,
132+
thing: builtins.str | None = ...,
133+
) -> None: ...
134+
def HasField(self, field_name: typing.Literal["thing", b"thing"]) -> builtins.bool: ...
135+
def ClearField(self, field_name: typing.Literal["thing", b"thing"]) -> None: ...
136+
137+
Global___Editions2024SubMessage: typing_extensions.TypeAlias = Editions2024SubMessage
138+
139+
@typing.final
140+
class Editions2024Test(google.protobuf.message.Message):
141+
DESCRIPTOR: google.protobuf.descriptor.Descriptor
142+
143+
LEGACY_FIELD_NUMBER: builtins.int
144+
EXPLICIT_SINGULAR_FIELD_NUMBER: builtins.int
145+
MESSAGE_FIELD_FIELD_NUMBER: builtins.int
146+
IMPLICIT_SINGULAR_FIELD_NUMBER: builtins.int
147+
DEFAULT_SINGULAR_FIELD_NUMBER: builtins.int
148+
legacy: builtins.str
149+
"""Expect to be always set"""
150+
explicit_singular: builtins.str
151+
"""Expect HasField generated"""
152+
implicit_singular: builtins.str
153+
"""Expect implicit field presence, no HasField generated"""
154+
default_singular: builtins.str
155+
"""Not set, should default to EXPLICIT"""
156+
@property
157+
def message_field(self) -> Global___Editions2024SubMessage:
158+
"""Expect HasField generated?"""
159+
160+
def __init__(
161+
self,
162+
*,
163+
legacy: builtins.str | None = ...,
164+
explicit_singular: builtins.str | None = ...,
165+
message_field: Global___Editions2024SubMessage | None = ...,
166+
implicit_singular: builtins.str = ...,
167+
default_singular: builtins.str | None = ...,
168+
) -> None: ...
169+
def HasField(self, field_name: typing.Literal["default_singular", b"default_singular", "explicit_singular", b"explicit_singular", "legacy", b"legacy", "message_field", b"message_field"]) -> builtins.bool: ...
170+
def ClearField(self, field_name: typing.Literal["default_singular", b"default_singular", "explicit_singular", b"explicit_singular", "implicit_singular", b"implicit_singular", "legacy", b"legacy", "message_field", b"message_field"]) -> None: ...
171+
172+
Global___Editions2024Test: typing_extensions.TypeAlias = Editions2024Test
173+
```
174+
175+
Builtin pyi generator:
176+
177+
```python
178+
from google.protobuf import descriptor as _descriptor
179+
from google.protobuf import message as _message
180+
from collections.abc import Mapping as _Mapping
181+
from typing import ClassVar as _ClassVar, Optional as _Optional, Union as _Union
182+
183+
DESCRIPTOR: _descriptor.FileDescriptor
184+
185+
class Editions2024SubMessage(_message.Message):
186+
__slots__ = ()
187+
THING_FIELD_NUMBER: _ClassVar[int]
188+
thing: str
189+
def __init__(self, thing: _Optional[str] = ...) -> None: ...
190+
191+
class Editions2024Test(_message.Message):
192+
__slots__ = ()
193+
LEGACY_FIELD_NUMBER: _ClassVar[int]
194+
EXPLICIT_SINGULAR_FIELD_NUMBER: _ClassVar[int]
195+
MESSAGE_FIELD_FIELD_NUMBER: _ClassVar[int]
196+
IMPLICIT_SINGULAR_FIELD_NUMBER: _ClassVar[int]
197+
DEFAULT_SINGULAR_FIELD_NUMBER: _ClassVar[int]
198+
legacy: str
199+
explicit_singular: str
200+
message_field: Editions2024SubMessage
201+
implicit_singular: str
202+
default_singular: str
203+
def __init__(self, legacy: _Optional[str] = ..., explicit_singular: _Optional[str] = ..., message_field: _Optional[_Union[Editions2024SubMessage, _Mapping]] = ..., implicit_singular: _Optional[str] = ..., default_singular: _Optional[str] = ...) -> None: ...
204+
```
205+
95206
### Bring comments from .proto files to docstrings in .pyi files
96207

97208
Comments in the .proto files on messages, fields, enums, enum variants, extensions, services, and methods

‎mypy_protobuf/extensions_pb2.py‎

Lines changed: 4 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎mypy_protobuf/main.py‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1073,8 +1073,6 @@ def write(self) -> str:
10731073
reexport_fd = self.descriptors.files[reexport_file]
10741074
reexport_imp = reexport_file[:-6].replace("-", "_").replace("/", ".") + "_pb2"
10751075
names = [m.name for m in reexport_fd.message_type] + [m.name for m in reexport_fd.enum_type] + [v.name for m in reexport_fd.enum_type for v in m.value] + [m.name for m in reexport_fd.extension]
1076-
if reexport_fd.options.py_generic_services:
1077-
names.extend(m.name for m in reexport_fd.service)
10781076

10791077
if names:
10801078
# n,n to force a reexport (from x import y as y)

‎proto/testproto/test.proto‎

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -165,17 +165,6 @@ service PythonReservedKeywordsService {
165165
rpc valid_method_name2(Simple1) returns (PythonReservedKeywords.lambda) {}
166166
}
167167

168-
// when service name itself is reserved - generated code was found to be invalid
169-
// in protoc 3.17.3
170-
//service global {
171-
// rpc Echo(Simple1) returns (Simple2) {}
172-
//}
173-
174-
option py_generic_services = true;
175-
service ATestService {
176-
rpc Echo(Simple1) returns (Simple2) {}
177-
}
178-
179168
message SelfField {
180169
// Field self -> must generate an __init__ method w/ different name
181170
optional int64 self = 1;

‎pyproject.toml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ report_deprecated_as_note = true
2323

2424
[tool.pyright]
2525
venvPath = "."
26+
venv = "venv_3.14.0"
2627
# verboseOutput = true
2728
extraPaths = ["test/generated"]
2829
include = [
@@ -33,3 +34,9 @@ exclude = [
3334
"**/*_pb2.py",
3435
"**/*_pb2_grpc.py"
3536
]
37+
38+
executionEnvironments = [
39+
# Due to how upb is typed, we need to disable incompatible variable override checks
40+
{ root = "test/generated", extraPaths = ["./"], reportIncompatibleVariableOverride = "none" },
41+
{ root = "mypy_protobuf/extensions_pb2.pyi", reportIncompatibleVariableOverride = "none" },
42+
]

‎run_test.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@
33
RED="\033[0;31m"
44
NC='\033[0m'
55

6-
PY_VER_MYPY_PROTOBUF=${PY_VER_MYPY_PROTOBUF:=3.14.0}
6+
PY_VER_MYPY_PROTOBUF=${PY_VER_MYPY_PROTOBUF:=3.12.12}
77
PY_VER_MYPY_PROTOBUF_SHORT=$(echo "$PY_VER_MYPY_PROTOBUF" | cut -d. -f1-2)
8-
PY_VER_MYPY=${PY_VER_MYPY:=3.14.0}
8+
PY_VER_MYPY=${PY_VER_MYPY:=3.12.12}
99
PY_VER_UNIT_TESTS="${PY_VER_UNIT_TESTS:=3.9.17 3.10.12 3.11.4 3.12.12 3.13.9 3.14.0}"
10-
PYTHON_PROTOBUF_VERSION=${PYTHON_PROTOBUF_VERSION:=5.28.3}
10+
PYTHON_PROTOBUF_VERSION=${PYTHON_PROTOBUF_VERSION:=6.32.1}
1111

1212
# Confirm UV installed
1313
if ! command -v uv &> /dev/null; then

‎stubtest_allowlist.txt‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,3 +150,86 @@ testproto.test_pb2.Simple1.InnerMessage
150150
testproto.test_pb2.Simple2
151151
testproto.test_pb2.DeprecatedMessage
152152
testproto.test_pb2.DeprecatedMessageBadComment
153+
154+
# All messages now fail with something like this:
155+
# error: testproto.nopackage_pb2.NoPackage2.DESCRIPTOR variable differs from runtime type google._upb._message.Descriptor
156+
# Stub: in file /Users/aidandj/Development/mypy-protobuf/test/generated/testproto/nopackage_pb2.pyi:35
157+
# google.protobuf.descriptor.Descriptor
158+
# Runtime:
159+
# <google._upb._message.Descriptor object at 0x1127a1f30>
160+
#
161+
# For now, just allow all these errors until we come up with better silencing mechanisms or
162+
# alternately, better stubs for google._upb
163+
testproto.Capitalized.Capitalized_pb2.Upper.DESCRIPTOR
164+
testproto.Capitalized.Capitalized_pb2.lower.DESCRIPTOR
165+
testproto.Capitalized.Capitalized_pb2.lower2.DESCRIPTOR
166+
testproto.comment_special_chars_pb2.Test.DESCRIPTOR
167+
testproto.dot.com.test_pb2.TestMessage.DESCRIPTOR
168+
testproto.grpc.dummy_pb2.DummyReply.DESCRIPTOR
169+
testproto.grpc.dummy_pb2.DummyRequest.DESCRIPTOR
170+
testproto.grpc.dummy_pb2.DeprecatedRequest.DESCRIPTOR
171+
testproto.inner.inner_pb2.Inner.DESCRIPTOR
172+
testproto.nested.nested_pb2.AnotherNested.DESCRIPTOR
173+
testproto.nested.nested_pb2.AnotherNested.NestedMessage.DESCRIPTOR
174+
testproto.nested.nested_pb2.Nested.DESCRIPTOR
175+
testproto.nopackage_pb2.NoPackage.DESCRIPTOR
176+
testproto.nopackage_pb2.NoPackage2.DESCRIPTOR
177+
testproto.reexport_pb2.Empty.DESCRIPTOR
178+
testproto.reexport_pb2.OuterMessage3.DESCRIPTOR
179+
testproto.reexport_pb2.SimpleProto3.DESCRIPTOR
180+
testproto.reexport_pb2.SimpleProto3.EmailByUidEntry.DESCRIPTOR
181+
testproto.reexport_pb2.SimpleProto3.MapMessageEntry.DESCRIPTOR
182+
testproto.reexport_pb2.SimpleProto3.MapScalarEntry.DESCRIPTOR
183+
testproto.test3_pb2.OuterMessage3.DESCRIPTOR
184+
testproto.test3_pb2.SimpleProto3.DESCRIPTOR
185+
testproto.test3_pb2.SimpleProto3.EmailByUidEntry.DESCRIPTOR
186+
testproto.test3_pb2.SimpleProto3.MapMessageEntry.DESCRIPTOR
187+
testproto.test3_pb2.SimpleProto3.MapScalarEntry.DESCRIPTOR
188+
testproto.test_extensions2_pb2.SeparateFileExtension.DESCRIPTOR
189+
testproto.test_extensions3_pb2.MessageOptionsTestMsg.DESCRIPTOR
190+
testproto.test_no_generic_services_pb2.Simple3.DESCRIPTOR
191+
testproto.test_pb2.Extensions1.DESCRIPTOR
192+
testproto.test_pb2.Extensions2.DESCRIPTOR
193+
testproto.test_pb2.SelfField.DESCRIPTOR
194+
testproto.test_pb2.Simple1.DESCRIPTOR
195+
testproto.test_pb2.Simple1.EmailByUidEntry.DESCRIPTOR
196+
testproto.test_pb2.Simple1.InnerMessage.DESCRIPTOR
197+
testproto.test_pb2.Simple2.DESCRIPTOR
198+
testproto.test_pb2.DeprecatedMessage.DESCRIPTOR
199+
testproto.test_pb2.DeprecatedMessageBadComment.DESCRIPTOR
200+
testproto.test_pb2.DESCRIPTOR
201+
testproto.test_no_generic_services_pb2.DESCRIPTOR
202+
testproto.test_extensions2_pb2.DESCRIPTOR
203+
testproto.test3_pb2.DESCRIPTOR
204+
testproto.reexport_pb2.DESCRIPTOR
205+
testproto.readme_enum_pb2.DESCRIPTOR
206+
testproto.nopackage_pb2.DESCRIPTOR
207+
testproto.nested.nested_pb2.DESCRIPTOR
208+
testproto.inner.inner_pb2.DESCRIPTOR
209+
testproto.grpc.import_pb2.DESCRIPTOR
210+
testproto.grpc.dummy_pb2.DESCRIPTOR
211+
testproto.dot.com.test_pb2.DESCRIPTOR
212+
testproto.comment_special_chars_pb2.DESCRIPTOR
213+
testproto.Capitalized.Capitalized_pb2.DESCRIPTOR
214+
testproto.test_extensions3_pb2.DESCRIPTOR
215+
216+
# All messages now fail with something like this:
217+
# error: testproto.test_extensions2_pb2.SeparateFileExtension.ext variable differs from runtime type google._upb._message.FieldDescriptor
218+
# Stub: in file /Users/aidandj/Development/mypy-protobuf/test/generated/testproto/test_extensions2_pb2.pyi:28
219+
# google.protobuf.internal.extension_dict._ExtensionFieldDescriptor[testproto.test_pb2.Simple2, testproto.test_extensions2_pb2.SeparateFileExtension]
220+
# Runtime:
221+
# <google._upb._message.FieldDescriptor object at 0x1045962f0>
222+
#
223+
# For now, just allow all these errors until we come up with better silencing mechanisms or
224+
# alternately, better stubs for google._upb
225+
testproto.test_pb2.Extensions2.foo
226+
testproto.test_pb2.Extensions1
227+
testproto.test_extensions3_pb2.test_field_extension
228+
testproto.test_extensions3_pb2.scalar_option
229+
testproto.test_extensions3_pb2.repeated_scalar_option
230+
testproto.test_extensions3_pb2.repeated_msg_option
231+
testproto.test_extensions3_pb2.repeated_enum_option
232+
testproto.test_extensions3_pb2.msg_option
233+
testproto.test_extensions3_pb2.enum_option
234+
testproto.test_extensions2_pb2.SeparateFileExtension.ext
235+
testproto.test_pb2.Extensions1.ext

0 commit comments

Comments
 (0)