The way `OpenAIChatModel` sends input audio is incompatible with Qwen Omni API

[mkroen]

Question

qwen omni
According to the Qwen Omni documentation, audio input must be formatted like this:

messages = [
  {
    "role": "user",
    "content": [
      {
        "input_audio": {
          "data": "data:;base64,{base64_audio}",
          "format": "wav"
        },
        "type": "input_audio"
      },
      {
        "text": "prompt",
        "type": "text"
      }
    ]
  }
]

I am using BinaryContent in Pydantic AI like this:

response = await agent.run(
    user_prompt=[
        BinaryContent(
            data=base64.b64decode(audio_base64),
            media_type="audio/wav",
        ),
        self.prompt,
    ],
)

But the serialized content becomes:

{"data": "{base64_audio}", "format": "wav"}

The generated output is missing the required prefix:

data:;base64,

This means the final payload is not compatible with Qwen Omni’s expected input format.

Question

Is there a correct way to:

1. Make BinaryContent output the audio in the required data:;base64,{...} format?
2. Or hook into the serialization layer so I can manually prepend the prefix?
3. Any recommended workaround or official way to customize the content part format would be helpful.

Additional Context

python==3.10.8
pydantic-ai-slim[openai]==1.21.0

LLM model: qwen3-omni-flash

[DouweM]

@mkroen The OpenAI SDK documents that the data field should be base64:

https://github.com/openai/openai-python/blob/41ee03ffe985a7362f0275c7f500080cb1d58cdd/src/openai/types/chat/chat_completion_content_part_input_audio_param.py#L10-L15

But based on the examples at https://help.aliyun.com/zh/model-studio/qwen-omni?mode=pure#c516d1e824x03, it looks like they've re-interpreted it as taking a URI instead so they can support both remote HTTP(S) URLs and inline data: URIs with base64 data, which makes sense but is incompatible with code that targets the official OpenAI API.

As a workaround, you can subclass OpenAIChatModel and override _map_user_prompt to post-process the generated ChatCompletionUserMessageParam typed dict which will have a contents list with, among other things, ChatCompletionContentPartInputAudioParam typed dicts which themselves will contain InputAudio typed dicts with the data field in question. Let me know if you'd like additional guidance on how to do that.

The "proper" way to support this would be to add a field to OpenAI's model profile: https://ai.pydantic.dev/models/openai/#model-profile so we can easily toggle the behavior. Feel free to submit a PR for that!

[Pavanmanikanta98]

Hi @DouweM,

I've reviewed the issue and your comments. I understand that Qwen Omni's audio input expects a Data URI format (data:;base64,...) which differs from the standard OpenAI raw base64.

Proposed Solution:
I agree with your suggestion to handle this via OpenAIModelProfile. I propose adding a new field:

openai_audio_input_encoding: Literal['base64', 'uri'] = 'base64'

In OpenAIChatModel._map_user_prompt, we can check this field:

• If 'base64' (default): Use existing logic (raw base64).
• If 'uri': Adjust to use item.data_uri for the 'uri' setting.

This would support Qwen Omni while preserving strict OpenAI compliance by default.

I'm happy to implement this PR if you'd like to ...

[DouweM]

@Pavanmanikanta98 Go for it!

[Pavanmanikanta98]

Hi @DouweM ,

I've implemented the fix as discussed! I've added openai_audio_input_encoding to OpenAIModelProfile to toggle between the default raw base64 format and the uri format required by Qwen Omni.
I've opened PR #3596 with the changes and tests verifying both behaviors.