diff --git a/docs/models/openai.md b/docs/models/openai.md index 2eae27bd03..c70d711859 100644 --- a/docs/models/openai.md +++ b/docs/models/openai.md @@ -210,6 +210,137 @@ print(result2.output) #> This is an excellent joke invented by Samuel Colvin, it needs no explanation. ``` +### Freeform Function Calling + +GPT‑5 can now send raw text payloads - anything from Python scripts to SQL queries - to your custom tool without wrapping the data in JSON using freeform function calling. This differs from classic structured function calls, giving you greater flexibility when interacting with external runtimes such as: + +* code execution with sandboxes (Python, C++, Java, …) +* SQL databases +* Shell environments +* Configuration generators + +Note that freeform function calling does NOT support parallel tool calling. + +You can enable freeform function calling for a tool by annotating the string parameter with [`FreeformText`][pydantic_ai.tools.FreeformText]. The tool must take a single string argument (other than the runtime context) and the model must be one of the GPT-5 responses models. For example: + +```python +from typing import Annotated + +from pydantic_ai import Agent, FreeformText +from pydantic_ai.models.openai import OpenAIResponsesModel + +model = OpenAIResponsesModel('gpt-5') # (1)! +agent = Agent(model) + +@agent.tool_plain +def freeform_tool(sql: Annotated[str, FreeformText()]): ... # (2)! +``` + +1. The GPT-5 family (`gpt-5`, `gpt-5-mini`, `gpt-5-nano`) all support freeform function calling. +2. If the tool or model cannot be used with freeform function calling then it will be invoked in the normal way. + +You can read more about this function calling style in the [OpenAI documentation](https://cookbook.openai.com/examples/gpt-5/gpt-5_new_params_and_tools#2-freeform-function-calling). + +#### Context Free Grammar + +A tool that queries an SQL database can only accept valid SQL. The freeform function calling of GPT-5 supports generation of valid SQL for this situation by constraining the generated text using a context free grammar. + +A context‑free grammar is a collection of production rules that define which strings belong to a language. Each rule rewrites a non‑terminal symbol into a sequence of terminals (literal tokens) and/or other non‑terminals, independent of surrounding context—hence context‑free. CFGs can capture the syntax of most programming languages and, in OpenAI custom tools, serve as contracts that force the model to emit only strings that the grammar accepts. + +##### Regular Expression + +The grammar can be written as either a regular expression using [`RegexGrammar`][pydantic_ai.tools.RegexGrammar]: + + +```python +from typing import Annotated + +from pydantic_ai import Agent, RegexGrammar +from pydantic_ai.models.openai import OpenAIResponsesModel + +model = OpenAIResponsesModel('gpt-5') # (1)! +agent = Agent(model) + +timestamp_pattern = r'^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]) (?:[01]\d|2[0-3]):[0-5]\d$' + +@agent.tool_plain +def timestamp_accepting_tool(timestamp: Annotated[str, RegexGrammar(timestamp_pattern)]): ... # (2)! +``` + +1. The GPT-5 family (`gpt-5`, `gpt-5-mini`, `gpt-5-nano`) all support freeform function calling with context free grammar constraints. Unfortunately `gpt-5-nano` often struggles with these calls. +2. If the tool or model cannot be used with freeform function calling then it will be invoked in the normal way, which may lead to invalid input. + +##### LARK + +Or as a [LARK](https://lark-parser.readthedocs.io/en/latest/how_to_use.html) grammar using [`LarkGrammar`][pydantic_ai.tools.LarkGrammar]: + +```python +from typing import Annotated + +from pydantic_ai import Agent, LarkGrammar +from pydantic_ai.models.openai import OpenAIResponsesModel + +model = OpenAIResponsesModel('gpt-5') # (1)! +agent = Agent(model) + +timestamp_grammar = r''' +start: timestamp + +timestamp: YEAR "-" MONTH "-" DAY " " HOUR ":" MINUTE + +%import common.DIGIT + +YEAR: DIGIT DIGIT DIGIT DIGIT +MONTH: /(0[1-9]|1[0-2])/ +DAY: /(0[1-9]|[12]\d|3[01])/ +HOUR: /([01]\d|2[0-3])/ +MINUTE: /[0-5]\d/ +''' + +@agent.tool_plain +def i_like_iso_dates(date: Annotated[str, LarkGrammar(timestamp_grammar)]): ... # (2)! +``` + +1. The GPT-5 family (`gpt-5`, `gpt-5-mini`, `gpt-5-nano`) all support freeform function calling with context free grammar constraints. Unfortunately `gpt-5-nano` often struggles with these calls. +2. If the tool or model cannot be used with freeform function calling then it will be invoked in the normal way, which may lead to invalid input. + +There is a limit to the grammar complexity that GPT-5 supports, as such it is important to test your grammar. + +Freeform function calling, with or without a context free grammar, can be used with the output type for the agent: + +```python +from typing import Annotated + +from pydantic_ai import Agent, LarkGrammar +from pydantic_ai.models.openai import OpenAIResponsesModel + +sql_grammar = r''' +start: select_stmt +select_stmt: "SELECT" select_list "FROM" table ("WHERE" condition ("AND" condition)*)? +select_list: "*" | column ("," column)* +table: "users" | "orders" +column: "id" | "user_id" | "name" | "age" +condition: column ("=" | ">" | "<") (NUMBER | STRING) +%import common.NUMBER +%import common.ESCAPED_STRING -> STRING +%import common.WS +%ignore WS +''' # (1)! + +model = OpenAIResponsesModel('gpt-5') +agent = Agent(model, output_type=Annotated[str, LarkGrammar(sql_grammar)]) +``` + +1. An inline SQL grammar definition would be quite extensive and so this simplified version has been written, you can find an example SQL grammar [in the openai example](https://cookbook.openai.com/examples/gpt-5/gpt-5_new_params_and_tools#33-example---sql-dialect--ms-sql-vs-postgresql). There are also example grammars in the [lark repo](https://github.com/lark-parser/lark/blob/master/examples/composition/json.lark). Remember that a simpler grammar that matches your DDL will be easier for GPT-5 to work with and will result in fewer semantically invalid results. + +##### Best Practices + +You can find recommended best practices in the [OpenAI Cookbook](https://cookbook.openai.com/examples/gpt-5/gpt-5_new_params_and_tools#35-best-practices). + +* [Lark Docs](https://lark-parser.readthedocs.io/en/stable/) +* [Lark IDE](https://www.lark-parser.org/ide/) +* [OpenAI Cookbook on CFG](https://cookbook.openai.com/examples/gpt-5/gpt-5_new_params_and_tools#3-contextfree-grammar-cfg) + ## OpenAI-compatible Models Many providers and models are compatible with the OpenAI API, and can be used with `OpenAIChatModel` in Pydantic AI. diff --git a/pydantic_ai_slim/pydantic_ai/__init__.py b/pydantic_ai_slim/pydantic_ai/__init__.py index c860d20dd8..c172527d5b 100644 --- a/pydantic_ai_slim/pydantic_ai/__init__.py +++ b/pydantic_ai_slim/pydantic_ai/__init__.py @@ -96,7 +96,19 @@ ) from .run import AgentRun, AgentRunResult, AgentRunResultEvent from .settings import ModelSettings -from .tools import DeferredToolRequests, DeferredToolResults, RunContext, Tool, ToolApproved, ToolDefinition, ToolDenied +from .tools import ( + DeferredToolRequests, + DeferredToolResults, + FreeformText, + LarkGrammar, + RegexGrammar, + RunContext, + TextFormat, + Tool, + ToolApproved, + ToolDefinition, + ToolDenied, +) from .toolsets import ( AbstractToolset, ApprovalRequiredToolset, @@ -201,6 +213,10 @@ 'DeferredToolResults', 'ToolApproved', 'ToolDenied', + 'TextFormat', + 'FreeformText', + 'RegexGrammar', + 'LarkGrammar', # toolsets 'AbstractToolset', 'ApprovalRequiredToolset', diff --git a/pydantic_ai_slim/pydantic_ai/_function_schema.py b/pydantic_ai_slim/pydantic_ai/_function_schema.py index 2b8270f322..772cc74b28 100644 --- a/pydantic_ai_slim/pydantic_ai/_function_schema.py +++ b/pydantic_ai_slim/pydantic_ai/_function_schema.py @@ -24,7 +24,7 @@ from ._utils import check_object_json_schema, is_async_callable, is_model_like, run_in_executor if TYPE_CHECKING: - from .tools import DocstringFormat, ObjectJsonSchema + from .tools import DocstringFormat, ObjectJsonSchema, TextFormat __all__ = ('function_schema',) @@ -44,6 +44,8 @@ class FunctionSchema: single_arg_name: str | None = None positional_fields: list[str] = field(default_factory=list) var_positional_field: str | None = None + text_format: TextFormat | None = None + """Text format annotation extracted from a string parameter, if present.""" async def call(self, args_dict: dict[str, Any], ctx: RunContext[Any]) -> Any: args, kwargs = self._call_args(args_dict, ctx) @@ -111,6 +113,7 @@ def function_schema( # noqa: C901 positional_fields: list[str] = [] var_positional_field: str | None = None decorators = _decorators.DecoratorInfos() + text_format: TextFormat | None = None description, field_descriptions = doc_descriptions(function, sig, docstring_format=docstring_format) @@ -147,6 +150,13 @@ def function_schema( # noqa: C901 errors.append('RunContext annotations can only be used as the first argument') continue + # Extract text format annotation if present + if extracted_format := _extract_text_format(annotation): + if text_format is not None: + errors.append('Only one parameter may have a TextFormat annotation') + else: + text_format = extracted_format + field_name = p.name if p.kind == Parameter.VAR_KEYWORD: var_kwargs_schema = gen_schema.generate_schema(annotation) @@ -222,6 +232,7 @@ def function_schema( # noqa: C901 takes_ctx=takes_ctx, is_async=is_async_callable(function), function=function, + text_format=text_format, ) @@ -301,3 +312,39 @@ def _build_schema( def _is_call_ctx(annotation: Any) -> bool: """Return whether the annotation is the `RunContext` class, parameterized or not.""" return annotation is RunContext or get_origin(annotation) is RunContext + + +def _extract_text_format(annotation: Any) -> TextFormat | None: + """Extract a TextFormat annotation from an Annotated type hint. + + Args: + annotation: The type annotation to check. + + Returns: + The TextFormat instance if found, None otherwise. + """ + from typing import Annotated, get_args, get_origin + + from .tools import FreeformText, LarkGrammar, RegexGrammar + + if get_origin(annotation) is not Annotated: + return None + + args = get_args(annotation) + if len(args) < 2: + return None + + # First arg is the base type, rest are metadata + base_type = args[0] + metadata = args[1:] + + # Check if base type is str + if base_type is not str: + return None + + # Look for TextFormat in metadata + for item in metadata: + if isinstance(item, (FreeformText, RegexGrammar, LarkGrammar)): + return item + + return None diff --git a/pydantic_ai_slim/pydantic_ai/_output.py b/pydantic_ai_slim/pydantic_ai/_output.py index e3adfbd190..abe02909e4 100644 --- a/pydantic_ai_slim/pydantic_ai/_output.py +++ b/pydantic_ai_slim/pydantic_ai/_output.py @@ -31,7 +31,7 @@ ToolOutput, _OutputSpecItem, # type: ignore[reportPrivateUsage] ) -from .tools import GenerateToolJsonSchema, ObjectJsonSchema, ToolDefinition +from .tools import GenerateToolJsonSchema, ObjectJsonSchema, TextFormat, ToolDefinition from .toolsets.abstract import AbstractToolset, ToolsetTool if TYPE_CHECKING: @@ -550,12 +550,17 @@ def __init__( description: str | None = None, strict: bool | None = None, ): + text_format: TextFormat | None = None + if inspect.isfunction(output) or inspect.ismethod(output): self._function_schema = _function_schema.function_schema(output, GenerateToolJsonSchema) self.validator = self._function_schema.validator json_schema = self._function_schema.json_schema json_schema['description'] = self._function_schema.description + text_format = self._function_schema.text_format else: + # Extract text_format from Annotated type if present + text_format = _function_schema._extract_text_format(output) json_schema_type_adapter: TypeAdapter[Any] validation_type_adapter: TypeAdapter[Any] if _utils.is_model_like(output): @@ -604,6 +609,7 @@ def __init__( description=description, json_schema=json_schema, strict=strict, + text_format=text_format, ) ) @@ -938,6 +944,7 @@ def build( description=description, parameters_json_schema=object_def.json_schema, strict=object_def.strict, + text_format=object_def.text_format, outer_typed_dict_key=processor.outer_typed_dict_key, kind='output', ) diff --git a/pydantic_ai_slim/pydantic_ai/models/openai.py b/pydantic_ai_slim/pydantic_ai/models/openai.py index a37be4f024..b0226e262e 100644 --- a/pydantic_ai_slim/pydantic_ai/models/openai.py +++ b/pydantic_ai_slim/pydantic_ai/models/openai.py @@ -51,7 +51,7 @@ from ..profiles.openai import OpenAIModelProfile, OpenAISystemPromptRole from ..providers import Provider, infer_provider from ..settings import ModelSettings -from ..tools import ToolDefinition +from ..tools import FreeformText, LarkGrammar, RegexGrammar, ToolDefinition from . import Model, ModelRequestParameters, StreamedResponse, check_allow_model_requests, download_item, get_user_agent try: @@ -90,6 +90,7 @@ from openai.types.responses.response_status import ResponseStatus from openai.types.shared import ReasoningEffort from openai.types.shared_params import Reasoning + from openai.types.shared_params.custom_tool_input_format import CustomToolInputFormat except ImportError as _import_error: raise ImportError( 'Please install `openai` to use the OpenAI model, ' @@ -1218,9 +1219,25 @@ def _process_response( # noqa: C901 elif isinstance(item, responses.ResponseComputerToolCall): # pragma: no cover # Pydantic AI doesn't yet support the ComputerUse built-in tool pass - elif isinstance(item, responses.ResponseCustomToolCall): # pragma: no cover - # Support is being implemented in https://github.com/pydantic/pydantic-ai/pull/2572 - pass + elif isinstance(item, responses.ResponseCustomToolCall): + # Handle custom tool calls (freeform function calling) + if item.name not in model_request_parameters.tool_defs: + argument_name = 'input' + else: + tool = model_request_parameters.tool_defs[item.name] + tool_argument_name = tool.single_string_argument_name + if tool_argument_name is None: + raise UnexpectedModelBehavior( + f'Custom tool call made to function {item.name} which has unexpected arguments' + ) + argument_name = tool_argument_name + items.append( + ToolCallPart( + item.name, + {argument_name: item.input}, + tool_call_id=_combine_tool_call_ids(item.call_id, item.id), + ) + ) elif isinstance(item, responses.response_output_item.LocalShellCall): # pragma: no cover # Pydantic AI doesn't yet support the `codex-mini-latest` LocalShell built-in tool pass @@ -1422,7 +1439,16 @@ def _get_reasoning(self, model_settings: OpenAIResponsesModelSettings) -> Reason return OMIT return Reasoning(effort=reasoning_effort, summary=reasoning_summary) - def _get_tools(self, model_request_parameters: ModelRequestParameters) -> list[responses.FunctionToolParam]: + def _get_parallel_tool_calling( + self, model_settings: OpenAIResponsesModelSettings, model_request_parameters: ModelRequestParameters + ) -> bool | NotGiven: + if any(tool_definition.text_format for tool_definition in model_request_parameters.tool_defs.values()): + return False + return model_settings.get('parallel_tool_calls', NOT_GIVEN) + + def _get_tools( + self, model_request_parameters: ModelRequestParameters + ) -> list[responses.FunctionToolParam | responses.CustomToolParam]: return [self._map_tool_definition(r) for r in model_request_parameters.tool_defs.values()] def _get_builtin_tools(self, model_request_parameters: ModelRequestParameters) -> list[responses.ToolParam]: @@ -1491,15 +1517,44 @@ def _get_builtin_tools(self, model_request_parameters: ModelRequestParameters) - tools.append({'type': 'image_generation'}) return tools - def _map_tool_definition(self, f: ToolDefinition) -> responses.FunctionToolParam: + def _map_tool_definition(self, f: ToolDefinition) -> responses.FunctionToolParam | responses.CustomToolParam: + model_profile = OpenAIModelProfile.from_profile(self.profile) + if f.text_format: + if not model_profile.openai_supports_freeform_function_calling: + raise UserError( + f'Tool {f.name!r} uses freeform function calling but {self._model_name!r} does not support freeform function calling.' + ) + if not f.only_takes_string_argument: + raise UserError( + f'`Tool {f.name!r}` is set as a freeform function but does not take a single string argument.' + ) + + # Handle different text format types + format: CustomToolInputFormat | None = None + if isinstance(f.text_format, FreeformText): + format = {'type': 'text'} + elif isinstance(f.text_format, RegexGrammar): + format = {'type': 'grammar', 'syntax': 'regex', 'definition': f.text_format.pattern} + elif isinstance(f.text_format, LarkGrammar): + format = {'type': 'grammar', 'syntax': 'lark', 'definition': f.text_format.definition} + + # If format was set (known type), return the custom tool param + # Otherwise fall through to return normal function tool (unknown text format type) + if format is not None: + tool_param: responses.CustomToolParam = { + 'name': f.name, + 'type': 'custom', + 'description': f.description or '', + 'format': format, + } + return tool_param + return { 'name': f.name, 'parameters': f.parameters_json_schema, 'type': 'function', 'description': f.description, - 'strict': bool( - f.strict and OpenAIModelProfile.from_profile(self.profile).openai_supports_strict_tool_definition - ), + 'strict': bool(f.strict and model_profile.openai_supports_strict_tool_definition), } def _get_previous_response_id_and_new_messages( @@ -2125,9 +2180,9 @@ async def _get_event_iterator(self) -> AsyncIterator[ModelResponseStreamEvent]: args_json = call_part.args_as_json_str() # Drop the final `{}}` so that we can add tool args deltas args_json_delta = args_json[:-3] - assert args_json_delta.endswith('"tool_args":'), ( - f'Expected {args_json_delta!r} to end in `"tool_args":"`' - ) + assert args_json_delta.endswith( + '"tool_args":' + ), f'Expected {args_json_delta!r} to end in `"tool_args":"`' yield self._parts_manager.handle_part( vendor_part_id=f'{chunk.item.id}-call', part=replace(call_part, args=None) @@ -2451,6 +2506,15 @@ def _map_provider_details( return provider_details +def _combine_tool_call_ids(call_id: str, id: str | None) -> str: + """Combine call_id and id into a single string for tool call tracking. + + When reasoning, the Responses API requires the `ResponseFunctionToolCall` to be returned with both the `call_id` and `id` fields. + Our `ToolCallPart` has only the `tool_call_id` field, so we combine the two fields into a single string. + """ + return f'{call_id}|{id}' if id else call_id + + def _split_combined_tool_call_id(combined_id: str) -> tuple[str, str | None]: # When reasoning, the Responses API requires the `ResponseFunctionToolCall` to be returned with both the `call_id` and `id` fields. # Before our `ToolCallPart` gained the `id` field alongside `tool_call_id` field, we combined the two fields into a single string stored on `tool_call_id`. diff --git a/pydantic_ai_slim/pydantic_ai/output.py b/pydantic_ai_slim/pydantic_ai/output.py index cd5e5865a6..88bcecd001 100644 --- a/pydantic_ai_slim/pydantic_ai/output.py +++ b/pydantic_ai_slim/pydantic_ai/output.py @@ -12,7 +12,7 @@ from . import _utils, exceptions from ._json_schema import InlineDefsJsonSchemaTransformer from .messages import ToolCallPart -from .tools import DeferredToolRequests, ObjectJsonSchema, RunContext, ToolDefinition +from .tools import DeferredToolRequests, ObjectJsonSchema, RunContext, TextFormat, ToolDefinition __all__ = ( # classes @@ -256,6 +256,7 @@ class OutputObjectDefinition: name: str | None = None description: str | None = None strict: bool | None = None + text_format: TextFormat | None = None @dataclass diff --git a/pydantic_ai_slim/pydantic_ai/profiles/openai.py b/pydantic_ai_slim/pydantic_ai/profiles/openai.py index 7bbc64c1a2..0325176da3 100644 --- a/pydantic_ai_slim/pydantic_ai/profiles/openai.py +++ b/pydantic_ai_slim/pydantic_ai/profiles/openai.py @@ -60,6 +60,11 @@ class OpenAIModelProfile(ModelProfile): openai_system_prompt_role: OpenAISystemPromptRole | None = None """The role to use for the system prompt message. If not provided, defaults to `'system'`.""" + # GPT-5 introduced support for directly calling a function with a string. + openai_supports_freeform_function_calling: bool = False + """Whether the provider accepts the value ``type='custom'`` for tools in the + request payload.""" + openai_chat_supports_web_search: bool = False """Whether the model supports web search in Chat Completions API.""" @@ -92,7 +97,7 @@ def openai_model_profile(model_name: str) -> ModelProfile: is_gpt_5 = model_name.startswith('gpt-5') is_o_series = model_name.startswith('o') is_reasoning_model = is_o_series or (is_gpt_5 and 'gpt-5-chat' not in model_name) - + is_freeform_function_calling_model = is_gpt_5 # Check if the model supports web search (only specific search-preview models) supports_web_search = '-search-preview' in model_name @@ -122,6 +127,7 @@ def openai_model_profile(model_name: str) -> ModelProfile: supports_json_schema_output=True, supports_json_object_output=True, supports_image_output=is_gpt_5 or 'o3' in model_name or '4.1' in model_name or '4o' in model_name, + openai_supports_freeform_function_calling=is_freeform_function_calling_model, openai_unsupported_model_settings=openai_unsupported_model_settings, openai_system_prompt_role=openai_system_prompt_role, openai_chat_supports_web_search=supports_web_search, diff --git a/pydantic_ai_slim/pydantic_ai/tools.py b/pydantic_ai_slim/pydantic_ai/tools.py index dcd860b019..c2b2f050a5 100644 --- a/pydantic_ai_slim/pydantic_ai/tools.py +++ b/pydantic_ai_slim/pydantic_ai/tools.py @@ -1,8 +1,10 @@ from __future__ import annotations as _annotations +import re from collections.abc import Awaitable, Callable, Sequence from dataclasses import KW_ONLY, dataclass, field from typing import Annotated, Any, Concatenate, Generic, Literal, TypeAlias, cast +from warnings import warn from pydantic import Discriminator, Tag from pydantic.json_schema import GenerateJsonSchema, JsonSchemaValue @@ -34,6 +36,10 @@ 'DeferredToolResults', 'ToolApproved', 'ToolDenied', + 'TextFormat', + 'FreeformText', + 'RegexGrammar', + 'LarkGrammar', ) @@ -229,6 +235,134 @@ class DeferredToolResults: A = TypeVar('A') +@dataclass +class FreeformText: + """Marker for freeform text input without grammar constraints. + + Use this annotation to indicate a tool parameter should receive raw text + input via freeform function calling, without any grammar constraints. + + Calling a function in this way prevents parallel tool calling. + + Example: + ```python + from typing import Annotated + + from pydantic_ai import Agent, FreeformText + + agent = Agent('openai:gpt-5') + + @agent.tool_plain + def run_code(code: Annotated[str, FreeformText()]) -> str: + return f'Executed: {code}' + ``` + + Note: this is currently only supported by OpenAI GPT-5 models. + """ + + +@dataclass +class RegexGrammar: + r"""Grammar constraint using regular expression pattern matching. + + Use this annotation to constrain tool input to match a regex pattern + via freeform function calling. + + Calling a function in this way prevents parallel tool calling. + + Example: + ```python + from typing import Annotated + + from pydantic_ai import Agent, RegexGrammar + + agent = Agent('openai:gpt-5') + + @agent.tool_plain + def parse_phone(phone: Annotated[str, RegexGrammar(r'\\d{3}-\\d{4}')]) -> str: + return f'Parsed phone: {phone}' + ``` + + Note: this is currently only supported by OpenAI GPT-5 models. + """ + + pattern: str + """The regular expression pattern that the text must conform to.""" + + def __post_init__(self) -> None: + try: + re.compile(self.pattern) + except re.error as e: + raise ValueError('Regex pattern is invalid') from e + + +@dataclass +class LarkGrammar: + """Grammar constraint using Lark parser grammar. + + Use this annotation to constrain tool input to match a Lark grammar + via freeform function calling. + + Requires the `lark` package to be installed for validation during tool definition. + + Calling a function in this way prevents parallel tool calling. + + Example: + ```python + from typing import Annotated + + from pydantic_ai import Agent, LarkGrammar + + agent = Agent('openai:gpt-5') + + grammar = ''' + start: "hello" name + name: /[A-Za-z]+/ + ''' + + @agent.tool_plain + def greet(text: Annotated[str, LarkGrammar(grammar)]) -> str: + return f'Greeting: {text}' + ``` + + Note: this is currently only supported by OpenAI GPT-5 models. + """ + + definition: str + """The Lark grammar definition that the text must conform to.""" + + def __post_init__(self) -> None: + try: + import lark + from lark.exceptions import GrammarError + + try: + lark.Lark(self.definition) + except GrammarError as e: + raise ValueError('Lark grammar is invalid') from e + except ImportError: + warn( + 'Cannot validate lark grammar as the lark optional dependency group has not been installed', + stacklevel=2, + ) # pragma: no cover + + +TextFormat: TypeAlias = FreeformText | RegexGrammar | LarkGrammar +"""Union of all supported text format types for freeform function calling. + +These types are used as annotations on string parameters to enable freeform function calling: + +- [`FreeformText`][pydantic_ai.tools.FreeformText]: Unconstrained text input +- [`RegexGrammar`][pydantic_ai.tools.RegexGrammar]: Text constrained by a regex pattern +- [`LarkGrammar`][pydantic_ai.tools.LarkGrammar]: Text constrained by a Lark grammar + +When a tool parameter is annotated with one of these types, the tool will use freeform +function calling instead of JSON-based function calling. This prevents parallel tool calling. + +Note: Support varies by model. Currently only OpenAI GPT-5 models support this feature. +""" + + class GenerateToolJsonSchema(GenerateJsonSchema): def typed_dict_schema(self, schema: core_schema.TypedDictSchema) -> JsonSchemaValue: json_schema = super().typed_dict_schema(schema) @@ -426,6 +560,7 @@ def tool_def(self): description=self.description, parameters_json_schema=self.function_schema.json_schema, strict=self.strict, + text_format=self.function_schema.text_format, sequential=self.sequential, metadata=self.metadata, kind='unapproved' if self.requires_approval else 'function', @@ -494,6 +629,18 @@ class ToolDefinition: Note: this is currently supported by OpenAI and Anthropic models. """ + text_format: TextFormat | None = None + """Text format annotation for freeform function calling. + + When set, the tool will use freeform function calling instead of JSON-based function calling. + This prevents parallel tool calling but allows passing raw text payloads to your custom tool + without wrapping the data in JSON. The function must take a single string argument. + + When `None` (the default), the model invokes the tool in the normal way and parallel tool calls are possible. + + Note: this is currently only supported by OpenAI GPT-5 models. + """ + sequential: bool = False """Whether this tool requires a sequential/serial execution environment.""" @@ -514,6 +661,28 @@ class ToolDefinition: For MCP tools, this contains the `meta`, `annotations`, and `output_schema` fields from the tool definition. """ + @property + def only_takes_string_argument(self) -> bool: + # true if the parameters_json_schema looks like: + # {"additionalProperties": False, "properties": {NAME: {"type": "string"}}, "required": ["NAME"], "type": "object"} + return self.single_string_argument_name is not None + + @property + def single_string_argument_name(self) -> str | None: + # returns the name of the single argument that is a string + # used for freeform function calling + # will return None if there is more or less than one argument, + # or if the argument is not a string + schema = self.parameters_json_schema + if len(schema['required']) != 1: + return None + if len(schema['properties']) != 1: + return None + property_name: str = schema['required'][0] + if not schema['properties'][property_name].get('type', None) == 'string': + return None + return property_name + @property def defer(self) -> bool: """Whether calls to this tool will be deferred. diff --git a/pydantic_ai_slim/pyproject.toml b/pydantic_ai_slim/pyproject.toml index ce1250f9f2..8a78150c4f 100644 --- a/pydantic_ai_slim/pyproject.toml +++ b/pydantic_ai_slim/pyproject.toml @@ -65,7 +65,7 @@ dependencies = [ [tool.hatch.metadata.hooks.uv-dynamic-versioning.optional-dependencies] # WARNING if you add optional groups, please update docs/install.md -logfire = ["logfire[httpx]>=3.14.1"] +logfire = ["logfire[httpx]>=3.16.1"] # Models openai = ["openai>=1.107.2"] cohere = ["cohere>=5.18.0; platform_system != 'Emscripten'"] @@ -112,6 +112,8 @@ temporal = ["temporalio==1.19.0"] dbos = ["dbos>=1.14.0"] # Prefect prefect = ["prefect>=3.4.21"] +# freeform function calling with lark context free grammar +lark = ["lark>=1.2.2"] [tool.hatch.metadata] allow-direct-references = true diff --git a/pyproject.toml b/pyproject.toml index 81754e81ef..a3ba2361a5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -46,7 +46,7 @@ requires-python = ">=3.10" [tool.hatch.metadata.hooks.uv-dynamic-versioning] dependencies = [ - "pydantic-ai-slim[openai,vertexai,google,groq,anthropic,mistral,cohere,bedrock,huggingface,cli,mcp,fastmcp,evals,ag-ui,retries,temporal,logfire,ui]=={{ version }}", + "pydantic-ai-slim[openai,vertexai,google,groq,anthropic,mistral,cohere,bedrock,huggingface,cli,mcp,fastmcp,evals,ag-ui,retries,temporal,logfire,ui,lark]=={{ version }}", ] [tool.hatch.metadata.hooks.uv-dynamic-versioning.optional-dependencies] diff --git a/tests/models/test_openai.py b/tests/models/test_openai.py index ed68edd94f..a7716d54e3 100644 --- a/tests/models/test_openai.py +++ b/tests/models/test_openai.py @@ -43,7 +43,7 @@ from pydantic_ai.profiles.openai import OpenAIModelProfile, openai_model_profile from pydantic_ai.result import RunUsage from pydantic_ai.settings import ModelSettings -from pydantic_ai.tools import ToolDefinition +from pydantic_ai.tools import FreeformText, LarkGrammar, RegexGrammar, ToolDefinition from pydantic_ai.usage import RequestUsage from ..conftest import IsDatetime, IsNow, IsStr, TestEnv, try_import @@ -2412,9 +2412,9 @@ def test_model_profile_strict_not_supported(): ) m = OpenAIChatModel('gpt-4o', provider=OpenAIProvider(api_key='foobar')) - tool_param = m._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + tool_definition = m._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] - assert tool_param == snapshot( + assert tool_definition == snapshot( { 'type': 'function', 'function': { @@ -2432,9 +2432,9 @@ def test_model_profile_strict_not_supported(): provider=OpenAIProvider(api_key='foobar'), profile=OpenAIModelProfile(openai_supports_strict_tool_definition=False).update(openai_model_profile('gpt-4o')), ) - tool_param = m._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + tool_definition = m._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] - assert tool_param == snapshot( + assert tool_definition == snapshot( { 'type': 'function', 'function': { @@ -3173,6 +3173,292 @@ def test_deprecated_openai_model(openai_api_key: str): OpenAIModel('gpt-4o', provider=provider) # type: ignore[reportDeprecated] +@pytest.mark.parametrize('model_name', ['gpt-5', 'gpt-5-mini', 'gpt-5-nano']) +def test_model_profile_gpt5_freeform_function_calling_support(model_name: str): + profile = cast('OpenAIModelProfile', openai_model_profile(model_name)) + assert profile.openai_supports_freeform_function_calling + + +@pytest.mark.parametrize('model_name', ['gpt-4.1', 'gpt-4o', 'gpt-o4-mini']) +def test_model_profile_gpt4_freeform_function_calling_support(model_name: str): + gpt4_profile = cast('OpenAIModelProfile', openai_model_profile(model_name)) + assert not gpt4_profile.openai_supports_freeform_function_calling + + +def test_chat_model_ignores_text_mode_text_when_tool_mapping(): + my_tool = ToolDefinition( + name='analyze_text', + description='Analyze the provided text', + parameters_json_schema={ + 'type': 'object', + 'properties': {'content': {'type': 'string'}}, + 'required': ['content'], + 'additionalProperties': False, + }, + text_format=FreeformText(), + ) + + model = OpenAIChatModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + tool_definition = model._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + + assert tool_definition == snapshot( + { + 'type': 'function', + 'function': { + 'name': 'analyze_text', + 'description': 'Analyze the provided text', + 'parameters': { + 'type': 'object', + 'properties': {'content': {'type': 'string'}}, + 'required': ['content'], + 'additionalProperties': False, + }, + }, + } + ) + + +def test_chat_model_ignores_text_mode_lark_when_tool_mapping(): + my_tool = ToolDefinition( + name='parse_data', + description='Parse structured data', + parameters_json_schema={ + 'type': 'object', + 'properties': {'data': {'type': 'string'}}, + 'required': ['data'], + 'additionalProperties': False, + }, + text_format=LarkGrammar(definition='start: "hello" " " "world"'), + ) + + model = OpenAIChatModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + tool_definition = model._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + + assert tool_definition == snapshot( + { + 'type': 'function', + 'function': { + 'name': 'parse_data', + 'description': 'Parse structured data', + 'parameters': { + 'type': 'object', + 'properties': {'data': {'type': 'string'}}, + 'required': ['data'], + 'additionalProperties': False, + }, + }, + } + ) + + +def test_chat_model_ignores_text_mode_regex_when_tool_mapping(): + my_tool = ToolDefinition( + name='extract_pattern', + description='Extract data matching pattern', + parameters_json_schema={ + 'type': 'object', + 'properties': {'text': {'type': 'string'}}, + 'required': ['text'], + 'additionalProperties': False, + }, + text_format=RegexGrammar(pattern=r'\d{4}-\d{2}-\d{2}'), + ) + + model = OpenAIChatModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + tool_definition = model._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + + assert tool_definition == snapshot( + { + 'type': 'function', + 'function': { + 'name': 'extract_pattern', + 'description': 'Extract data matching pattern', + 'parameters': { + 'type': 'object', + 'properties': {'text': {'type': 'string'}}, + 'required': ['text'], + 'additionalProperties': False, + }, + }, + } + ) + + +def test_responses_model_uses_text_mode_freeform_when_tool_mapping(): + my_tool = ToolDefinition( + name='analyze_text', + description='Analyze the provided text', + parameters_json_schema={ + 'type': 'object', + 'properties': {'content': {'type': 'string'}}, + 'required': ['content'], + 'additionalProperties': False, + }, + text_format=FreeformText(), + ) + + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + tool_definition = model._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + + assert tool_definition == snapshot( + { + 'name': 'analyze_text', + 'type': 'custom', + 'description': 'Analyze the provided text', + 'format': {'type': 'text'}, + } + ) + + +def test_responses_model_uses_text_mode_lark_when_tool_mapping(): + my_tool = ToolDefinition( + name='parse_data', + description='Parse structured data', + parameters_json_schema={ + 'type': 'object', + 'properties': {'data': {'type': 'string'}}, + 'required': ['data'], + 'additionalProperties': False, + }, + text_format=LarkGrammar(definition='start: "hello" " " "world"'), + ) + + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + tool_definition = model._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + + assert tool_definition == snapshot( + { + 'name': 'parse_data', + 'type': 'custom', + 'description': 'Parse structured data', + 'format': {'type': 'grammar', 'syntax': 'lark', 'definition': 'start: "hello" " " "world"'}, + } + ) + + +def test_responses_model_uses_text_mode_regex_when_tool_mapping(): + my_tool = ToolDefinition( + name='extract_pattern', + description='Extract data matching pattern', + parameters_json_schema={ + 'type': 'object', + 'properties': {'text': {'type': 'string'}}, + 'required': ['text'], + 'additionalProperties': False, + }, + text_format=RegexGrammar(pattern=r'\d{4}-\d{2}-\d{2}'), + ) + + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + tool_definition = model._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + + assert tool_definition == snapshot( + { + 'name': 'extract_pattern', + 'type': 'custom', + 'description': 'Extract data matching pattern', + 'format': {'type': 'grammar', 'syntax': 'regex', 'definition': '\\d{4}-\\d{2}-\\d{2}'}, + } + ) + + +def test_chat_model_tool_mapping_regular_function_unchanged(): + my_tool = ToolDefinition( + name='regular_tool', + description='A regular tool', + parameters_json_schema={ + 'type': 'object', + 'properties': {'param': {'type': 'string'}}, + 'required': ['param'], + }, + ) + + model = OpenAIChatModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + tool_definition = model._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + + assert tool_definition == snapshot( + { + 'type': 'function', + 'function': { + 'name': 'regular_tool', + 'description': 'A regular tool', + 'parameters': {'type': 'object', 'properties': {'param': {'type': 'string'}}, 'required': ['param']}, + }, + } + ) + + +def test_responses_model_tool_mapping_regular_function_unchanged(): + my_tool = ToolDefinition( + name='regular_tool', + description='A regular tool', + parameters_json_schema={ + 'type': 'object', + 'properties': {'param': {'type': 'string'}}, + 'required': ['param'], + }, + ) + + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + tool_definition = model._map_tool_definition(my_tool) # type: ignore[reportPrivateUsage] + + assert tool_definition == snapshot( + { + 'name': 'regular_tool', + 'parameters': {'type': 'object', 'properties': {'param': {'type': 'string'}}, 'required': ['param']}, + 'type': 'function', + 'description': 'A regular tool', + 'strict': False, + } + ) + + +def test_tool_definition_single_string_argument(): + valid_tool = ToolDefinition( + name='valid_tool', + description='Valid single string tool', + parameters_json_schema={ + 'type': 'object', + 'properties': {'content': {'type': 'string'}}, + 'required': ['content'], + 'additionalProperties': False, + }, + ) + assert valid_tool.only_takes_string_argument + assert valid_tool.single_string_argument_name == 'content' + + +def test_tool_definition_multiple_argument_single_string_argument(): + multi_param_tool = ToolDefinition( + name='multi_tool', + description='Multi param tool', + parameters_json_schema={ + 'type': 'object', + 'param1': {'type': 'string'}, + 'properties': { + 'param2': {'type': 'string'}, + }, + 'required': ['param1', 'param2'], + }, + ) + assert not multi_param_tool.only_takes_string_argument + assert multi_param_tool.single_string_argument_name is None + + +def test_tool_definition_single_non_string_argument_single_string_argument(): + non_string_tool = ToolDefinition( + name='non_string_tool', + description='Non-string param tool', + parameters_json_schema={ + 'type': 'object', + 'properties': {'count': {'type': 'integer'}}, + 'required': ['count'], + }, + ) + assert not non_string_tool.only_takes_string_argument + assert non_string_tool.single_string_argument_name is None + + async def test_cache_point_filtering(allow_model_requests: None): """Test that CachePoint is filtered out in OpenAI Chat Completions requests.""" c = completion_message(ChatCompletionMessage(content='response', role='assistant')) diff --git a/tests/models/test_openai_responses.py b/tests/models/test_openai_responses.py index 47bdeaea3f..767920d822 100644 --- a/tests/models/test_openai_responses.py +++ b/tests/models/test_openai_responses.py @@ -1,6 +1,7 @@ import json import re from dataclasses import replace +from datetime import datetime, timezone from typing import Any, cast import pytest @@ -37,7 +38,7 @@ ) from pydantic_ai.agent import Agent from pydantic_ai.builtin_tools import CodeExecutionTool, MCPServerTool, WebSearchTool -from pydantic_ai.exceptions import ModelHTTPError, ModelRetry +from pydantic_ai.exceptions import ModelHTTPError, ModelRetry, UserError from pydantic_ai.messages import ( BuiltinToolCallEvent, # pyright: ignore[reportDeprecated] BuiltinToolResultEvent, # pyright: ignore[reportDeprecated] @@ -45,13 +46,15 @@ from pydantic_ai.models import ModelRequestParameters from pydantic_ai.output import NativeOutput, PromptedOutput, TextOutput, ToolOutput from pydantic_ai.profiles.openai import openai_model_profile -from pydantic_ai.tools import ToolDefinition +from pydantic_ai.tools import FreeformText, ToolDefinition from pydantic_ai.usage import RequestUsage, RunUsage from ..conftest import IsBytes, IsDatetime, IsStr, TestEnv, try_import from .mock_openai import MockOpenAIResponses, get_mock_responses_kwargs, response_message with try_import() as imports_successful: + from openai import NOT_GIVEN + from openai.types.responses.response_output_item import ResponseCustomToolCall from openai.types.responses.response_output_message import Content, ResponseOutputMessage, ResponseOutputText from openai.types.responses.response_reasoning_item import ( Content as ReasoningContent, @@ -2034,6 +2037,223 @@ async def test_openai_responses_usage_without_tokens_details(allow_model_request ) +def test_openai_responses_model_parallel_tool_calling_enabled(): + regular_tool = ToolDefinition( + name='regular_function', + description='A regular function', + parameters_json_schema={ + 'type': 'object', + 'properties': {'param': {'type': 'string'}}, + 'required': ['param'], + }, + ) + + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + + params_regular_only = ModelRequestParameters(function_tools=[regular_tool]) + parallel_calling = model._get_parallel_tool_calling({}, params_regular_only) # type: ignore[reportPrivateUsage] + assert parallel_calling == NOT_GIVEN + + +def test_openai_responses_model_parallel_tool_calling_disabled_with_freeform(): + freeform_tool = ToolDefinition( + name='freeform_analyzer', + description='A freeform analyzer', + parameters_json_schema={ + 'type': 'object', + 'properties': {'content': {'type': 'string'}}, + 'required': ['content'], + }, + text_format=FreeformText(), + ) + + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + + params_with_freeform = ModelRequestParameters(function_tools=[freeform_tool]) + parallel_calling = model._get_parallel_tool_calling({}, params_with_freeform) # type: ignore[reportPrivateUsage] + assert not parallel_calling + + +def test_openai_responses_model_parallel_tool_calling_disabled_with_freeform_output(): + freeform_tool = ToolDefinition( + name='freeform_analyzer', + description='A freeform analyzer', + parameters_json_schema={ + 'type': 'object', + 'properties': {'content': {'type': 'string'}}, + 'required': ['content'], + }, + text_format=FreeformText(), + ) + + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + + params_with_freeform = ModelRequestParameters(output_tools=[freeform_tool]) + parallel_calling = model._get_parallel_tool_calling({}, params_with_freeform) # type: ignore[reportPrivateUsage] + assert not parallel_calling + + +def test_openai_responses_model_freeform_function_unsupported_model_error(): + freeform_tool = ToolDefinition( + name='freeform_analyzer', + description='A freeform analyzer', + parameters_json_schema={ + 'type': 'object', + 'properties': {'content': {'type': 'string'}}, + 'required': ['content'], + }, + text_format=FreeformText(), + ) + + # GPT-4 doesn't support freeform function calling + model = OpenAIResponsesModel('gpt-4o', provider=OpenAIProvider(api_key='foobar')) + + with pytest.raises( + UserError, match=r'uses freeform function calling but .* does not support freeform function calling' + ): + model._map_tool_definition(freeform_tool) # type: ignore[reportPrivateUsage] + + +def test_openai_responses_model_freeform_function_invalid_signature_error(): + multi_param_tool = ToolDefinition( + name='multi_param_analyzer', + description='Tool with multiple params', + parameters_json_schema={ + 'type': 'object', + 'properties': { + 'param1': {'type': 'string'}, + 'param2': {'type': 'string'}, + }, + 'required': ['param1', 'param2'], + }, + text_format=FreeformText(), + ) + + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key='foobar')) + + with pytest.raises(UserError, match=r'is set as a freeform function but does not take a single string argument'): + model._map_tool_definition(multi_param_tool) # type: ignore[reportPrivateUsage] + + +async def test_openai_responses_model_custom_tool_call_response_processing(allow_model_requests: None): + """Test that OpenAI Responses model processes custom_tool_call responses correctly.""" + from pydantic_ai.models import ModelRequestParameters + + content_data = [ + ResponseCustomToolCall( + type='custom_tool_call', + name='analyze_content', + call_id='call_custom_456', + input='This is the raw content input', + ) + ] + + mock_response = response_message(content_data) + mock_client = MockOpenAIResponses.create_mock(mock_response) + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(openai_client=mock_client)) + + freeform_tool = ToolDefinition( + name='analyze_content', + description='Analyze content', + parameters_json_schema={ + 'type': 'object', + 'properties': {'content': {'type': 'string'}}, + 'required': ['content'], + }, + text_format=FreeformText(), + ) + + params = ModelRequestParameters(function_tools=[freeform_tool]) + + response = model._process_response(mock_response, params) # type: ignore[reportPrivateUsage] + + assert response == snapshot( + ModelResponse( + parts=[ + ToolCallPart( + tool_name='analyze_content', + args={'content': 'This is the raw content input'}, + tool_call_id='call_custom_456', + ) + ], + model_name='gpt-4o-123', + timestamp=datetime(2024, 1, 1, 0, 0, tzinfo=timezone.utc), + provider_name='openai', + provider_response_id='123', + ) + ) + + +async def test_openai_responses_model_custom_tool_call_unknown_tool_parsed(allow_model_requests: None): + """Test that unknown custom tool calls are parsed into ToolCallPart with 'input' as argument name. + + Unknown tools are handled during execution (not response processing) per the architecture pattern. + """ + from pydantic_ai.models import ModelRequestParameters + + content_data = [ + ResponseCustomToolCall( + type='custom_tool_call', + name='unknown_analyzer', + call_id='call_unknown_456', + input='Some content', + ) + ] + + mock_response = response_message(content_data) + mock_client = MockOpenAIResponses.create_mock(mock_response) + m = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(openai_client=mock_client)) + + params = ModelRequestParameters() # No tools defined + + # Should not raise an error - unknown tools are handled during execution + response = m._process_response(mock_response, params) # type: ignore[reportPrivateUsage] + + # Verify that a ToolCallPart was created with 'input' as the default argument name + assert len(response.parts) == 1 + tool_call = response.parts[0] + assert isinstance(tool_call, ToolCallPart) + assert tool_call.tool_name == 'unknown_analyzer' + assert tool_call.args == {'input': 'Some content'} + assert tool_call.tool_call_id == 'call_unknown_456' + + +async def test_openai_responses_model_custom_tool_call_invalid_signature_error(allow_model_requests: None): + """Test that OpenAI Responses model raises error for custom tool calls to tools with invalid signatures.""" + from pydantic_ai.models import ModelRequestParameters + + content_data = [ + ResponseCustomToolCall( + type='custom_tool_call', + name='invalid_analyzer', + call_id='call_invalid_456', + input='Some content', + ) + ] + + mock_response = response_message(content_data) + mock_client = MockOpenAIResponses.create_mock(mock_response) + model = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(openai_client=mock_client)) + + invalid_tool = ToolDefinition( + name='invalid_analyzer', + description='Tool with invalid signature', + parameters_json_schema={ + 'type': 'object', + 'properties': { + 'param1': {'type': 'string'}, + 'param2': {'type': 'string'}, + }, + 'required': ['param1', 'param2'], + }, + ) + + params = ModelRequestParameters(function_tools=[invalid_tool]) + + with pytest.raises(UnexpectedModelBehavior, match='has unexpected arguments'): + model._process_response(mock_response, params) # type: ignore[reportPrivateUsage] + + async def test_openai_responses_model_thinking_part(allow_model_requests: None, openai_api_key: str): m = OpenAIResponsesModel('gpt-5', provider=OpenAIProvider(api_key=openai_api_key)) settings = OpenAIResponsesModelSettings(openai_reasoning_effort='high', openai_reasoning_summary='detailed') diff --git a/tests/test_logfire.py b/tests/test_logfire.py index dadb930dd0..5a1a076da2 100644 --- a/tests/test_logfire.py +++ b/tests/test_logfire.py @@ -544,6 +544,7 @@ async def my_ret(x: int) -> str: }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'sequential': False, 'kind': 'function', 'metadata': None, @@ -991,6 +992,7 @@ class MyOutput: 'description': 'The final response which ends this conversation', 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'sequential': False, 'kind': 'output', 'metadata': None, diff --git a/tests/test_tools.py b/tests/test_tools.py index bcdf537994..6dc64d1b76 100644 --- a/tests/test_tools.py +++ b/tests/test_tools.py @@ -1,3 +1,4 @@ +import importlib.util import json import re from collections.abc import Callable @@ -35,7 +36,13 @@ from pydantic_ai.models.function import AgentInfo, FunctionModel from pydantic_ai.models.test import TestModel from pydantic_ai.output import ToolOutput -from pydantic_ai.tools import DeferredToolRequests, DeferredToolResults, ToolApproved, ToolDefinition, ToolDenied +from pydantic_ai.tools import ( + DeferredToolRequests, + DeferredToolResults, + ToolApproved, + ToolDefinition, + ToolDenied, +) from pydantic_ai.usage import RequestUsage from .conftest import IsDatetime, IsStr @@ -147,6 +154,7 @@ def test_docstring_google(docstring_format: Literal['google', 'auto']): }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -181,6 +189,7 @@ def test_docstring_sphinx(docstring_format: Literal['sphinx', 'auto']): }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -223,6 +232,7 @@ def test_docstring_numpy(docstring_format: Literal['numpy', 'auto']): }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -265,6 +275,7 @@ def my_tool(x: int) -> str: # pragma: no cover }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -305,6 +316,7 @@ def my_tool(x: int) -> str: # pragma: no cover }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -351,6 +363,7 @@ def my_tool(x: int) -> str: # pragma: no cover }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -385,6 +398,7 @@ def test_only_returns_type(): 'parameters_json_schema': {'additionalProperties': False, 'properties': {}, 'type': 'object'}, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -410,6 +424,7 @@ def test_docstring_unknown(): 'parameters_json_schema': {'additionalProperties': {'type': 'integer'}, 'properties': {}, 'type': 'object'}, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -453,6 +468,7 @@ def test_docstring_google_no_body(docstring_format: Literal['google', 'auto']): }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -489,6 +505,7 @@ def takes_just_model(model: Foo) -> str: }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -534,6 +551,7 @@ def takes_just_model(model: Foo, z: int) -> str: }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -899,6 +917,7 @@ def test_suppress_griffe_logging(caplog: LogCaptureFixture): 'outer_typed_dict_key': None, 'parameters_json_schema': {'additionalProperties': False, 'properties': {}, 'type': 'object'}, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -971,6 +990,7 @@ def my_tool_plain(*, a: int = 1, b: int) -> int: 'type': 'object', }, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -986,6 +1006,7 @@ def my_tool_plain(*, a: int = 1, b: int) -> int: 'type': 'object', }, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -1074,6 +1095,7 @@ def my_tool(x: Annotated[str | None, WithJsonSchema({'type': 'string'})] = None, 'type': 'object', }, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -1087,6 +1109,7 @@ def my_tool(x: Annotated[str | None, WithJsonSchema({'type': 'string'})] = None, 'type': 'object', }, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -1124,6 +1147,7 @@ def get_score(data: Data) -> int: ... # pragma: no branch }, 'outer_typed_dict_key': None, 'strict': None, + 'text_format': None, 'kind': 'function', 'sequential': False, 'metadata': None, @@ -2073,6 +2097,134 @@ def buy(fruit: str): ) +def test_regex_grammar_valid(): + from pydantic_ai.tools import RegexGrammar + + grammar = RegexGrammar(pattern=r'\d+') + assert grammar.pattern == r'\d+' + + +def test_regex_grammar_invalid(): + from pydantic_ai.tools import RegexGrammar + + with pytest.raises(ValueError, match='Regex pattern is invalid'): + RegexGrammar(pattern='[') + + +@pytest.mark.skipif(not importlib.util.find_spec('lark'), reason='lark not installed') +def test_lark_grammar_valid(): + from pydantic_ai.tools import LarkGrammar + + grammar = LarkGrammar(definition='start: "hello"') + assert grammar.definition == 'start: "hello"' + + +@pytest.mark.skipif(not importlib.util.find_spec('lark'), reason='lark not installed') +def test_lark_grammar_invalid(): + from pydantic_ai.tools import LarkGrammar + + with pytest.raises(ValueError, match='Lark grammar is invalid'): + LarkGrammar(definition='invalid grammar [') + + +def test_tool_definition_single_string_argument(): + schema = { + 'type': 'object', + 'properties': {'text': {'type': 'string'}}, + 'required': ['text'], + 'additionalProperties': False, + } + tool_def = ToolDefinition(name='test', parameters_json_schema=schema) + assert tool_def.single_string_argument_name == 'text' + assert tool_def.only_takes_string_argument + + +def test_tool_definition_multiple_arguments(): + schema = { + 'type': 'object', + 'properties': {'text': {'type': 'string'}, 'count': {'type': 'integer'}}, + 'required': ['text', 'count'], + 'additionalProperties': False, + } + tool_def = ToolDefinition(name='test', parameters_json_schema=schema) + assert tool_def.single_string_argument_name is None + assert not tool_def.only_takes_string_argument + + +def test_tool_definition_non_string_argument(): + schema = { + 'type': 'object', + 'properties': {'count': {'type': 'integer'}}, + 'required': ['count'], + 'additionalProperties': False, + } + tool_def = ToolDefinition(name='test', parameters_json_schema=schema) + assert tool_def.single_string_argument_name is None + assert not tool_def.only_takes_string_argument + + +def test_tool_definition_no_required_fields(): + required: list[str] = [] + schema = { + 'type': 'object', + 'properties': {'text': {'type': 'string'}}, + 'required': required, + 'additionalProperties': False, + } + tool_def = ToolDefinition(name='test', parameters_json_schema=schema) + assert tool_def.single_string_argument_name is None + assert not tool_def.only_takes_string_argument + + +def test_tool_definition_no_properties(): + required: list[str] = [] + properties: dict[str, dict[str, str]] = {} + schema = {'type': 'object', 'properties': properties, 'required': required, 'additionalProperties': False} + tool_def = ToolDefinition(name='test', parameters_json_schema=schema) + assert tool_def.single_string_argument_name is None + assert not tool_def.only_takes_string_argument + + +def test_tool_definition_mismatched_properties_required(): + schema = { + 'type': 'object', + 'properties': {'text': {'type': 'string'}, 'extra': {'type': 'string'}}, + 'required': ['text'], + 'additionalProperties': False, + } + tool_def = ToolDefinition(name='test', parameters_json_schema=schema) + assert tool_def.single_string_argument_name is None + assert not tool_def.only_takes_string_argument + + +def test_agent_tool_with_freeform_text(): + from pydantic_ai.tools import FreeformText + + agent = Agent(TestModel()) + + @agent.tool_plain + def analyze_text(text: Annotated[str, FreeformText()]) -> str: + return f'Analyzed: {text}' # pragma: no cover + + tool_def = agent._function_toolset.tools['analyze_text'].tool_def + assert isinstance(tool_def.text_format, FreeformText) + assert tool_def.only_takes_string_argument + + +def test_agent_tool_with_regex_grammar(): + from pydantic_ai.tools import RegexGrammar + + agent = Agent(TestModel()) + + @agent.tool_plain + def parse_numbers(numbers: Annotated[str, RegexGrammar(r'\d+')]) -> str: + return f'Parsed: {numbers}' # pragma: no cover + + tool_def = agent._function_toolset.tools['parse_numbers'].tool_def + assert isinstance(tool_def.text_format, RegexGrammar) + assert tool_def.text_format.pattern == r'\d+' + + def test_deferred_tool_call_approved_fails(): def llm(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse: return ModelResponse( diff --git a/uv.lock b/uv.lock index 0079a4a9c5..403008b9af 100644 --- a/uv.lock +++ b/uv.lock @@ -5365,7 +5365,7 @@ email = [ name = "pydantic-ai" source = { editable = "." } dependencies = [ - { name = "pydantic-ai-slim", extra = ["ag-ui", "anthropic", "bedrock", "cli", "cohere", "evals", "fastmcp", "google", "groq", "huggingface", "logfire", "mcp", "mistral", "openai", "retries", "temporal", "ui", "vertexai"] }, + { name = "pydantic-ai-slim", extra = ["ag-ui", "anthropic", "bedrock", "cli", "cohere", "evals", "fastmcp", "google", "groq", "huggingface", "lark", "logfire", "mcp", "mistral", "openai", "retries", "temporal", "ui", "vertexai"] }, ] [package.optional-dependencies] @@ -5445,7 +5445,7 @@ lint = [ requires-dist = [ { name = "fasta2a", marker = "extra == 'a2a'", specifier = ">=0.4.1" }, { name = "pydantic-ai-examples", marker = "extra == 'examples'", editable = "examples" }, - { name = "pydantic-ai-slim", extras = ["ag-ui", "anthropic", "bedrock", "cli", "cohere", "evals", "fastmcp", "google", "groq", "huggingface", "logfire", "mcp", "mistral", "openai", "retries", "temporal", "ui", "vertexai"], editable = "pydantic_ai_slim" }, + { name = "pydantic-ai-slim", extras = ["ag-ui", "anthropic", "bedrock", "cli", "cohere", "evals", "fastmcp", "google", "groq", "huggingface", "lark", "logfire", "mcp", "mistral", "openai", "retries", "temporal", "ui", "vertexai"], editable = "pydantic_ai_slim" }, { name = "pydantic-ai-slim", extras = ["dbos"], marker = "extra == 'dbos'", editable = "pydantic_ai_slim" }, { name = "pydantic-ai-slim", extras = ["outlines-llamacpp"], marker = "extra == 'outlines-llamacpp'", editable = "pydantic_ai_slim" }, { name = "pydantic-ai-slim", extras = ["outlines-mlxlm"], marker = "platform_machine == 'arm64' and sys_platform == 'darwin' and extra == 'outlines-mlxlm'", editable = "pydantic_ai_slim" }, @@ -5599,6 +5599,9 @@ huggingface = [ { name = "huggingface-hub", version = "0.33.5", source = { registry = "https://pypi.org/simple" }, extra = ["inference"], marker = "python_full_version >= '3.12'" }, { name = "huggingface-hub", version = "0.36.0", source = { registry = "https://pypi.org/simple" }, extra = ["inference"], marker = "python_full_version < '3.12'" }, ] +lark = [ + { name = "lark" }, +] logfire = [ { name = "logfire", extra = ["httpx"] }, ] @@ -5677,7 +5680,8 @@ requires-dist = [ { name = "groq", marker = "extra == 'groq'", specifier = ">=0.25.0" }, { name = "httpx", specifier = ">=0.27" }, { name = "huggingface-hub", extras = ["inference"], marker = "extra == 'huggingface'", specifier = ">=0.33.5,<1.0.0" }, - { name = "logfire", extras = ["httpx"], marker = "extra == 'logfire'", specifier = ">=3.14.1" }, + { name = "lark", marker = "extra == 'lark'", specifier = ">=1.2.2" }, + { name = "logfire", extras = ["httpx"], marker = "extra == 'logfire'", specifier = ">=3.16.1" }, { name = "mcp", marker = "extra == 'mcp'", specifier = ">=1.18.0" }, { name = "mistralai", marker = "extra == 'mistral'", specifier = ">=1.9.10" }, { name = "openai", marker = "extra == 'openai'", specifier = ">=1.107.2" }, @@ -5709,7 +5713,7 @@ requires-dist = [ { name = "typing-inspection", specifier = ">=0.4.0" }, { name = "vllm", marker = "(python_full_version < '3.12' and platform_machine != 'x86_64' and extra == 'outlines-vllm-offline') or (python_full_version < '3.12' and sys_platform != 'darwin' and extra == 'outlines-vllm-offline')" }, ] -provides-extras = ["a2a", "ag-ui", "anthropic", "bedrock", "cli", "cohere", "dbos", "duckduckgo", "evals", "fastmcp", "google", "groq", "huggingface", "logfire", "mcp", "mistral", "openai", "openrouter", "outlines-llamacpp", "outlines-mlxlm", "outlines-sglang", "outlines-transformers", "outlines-vllm-offline", "prefect", "retries", "tavily", "temporal", "ui", "vertexai"] +provides-extras = ["a2a", "ag-ui", "anthropic", "bedrock", "cli", "cohere", "dbos", "duckduckgo", "evals", "fastmcp", "google", "groq", "huggingface", "lark", "logfire", "mcp", "mistral", "openai", "openrouter", "outlines-llamacpp", "outlines-mlxlm", "outlines-sglang", "outlines-transformers", "outlines-vllm-offline", "prefect", "retries", "tavily", "temporal", "ui", "vertexai"] [[package]] name = "pydantic-core"