Support tool calling in Langchain OpenAI client

[alexbau01]

Describe the Problem

Native Tool calling (or function calling) provides several advantages, including making it easier to build AI Agents that request data from external systems as well as generating structured output from an LLM.

In my testing, the Python SDK supports tool calling, while the JavaScript SDK does not.

I am happy to close the issue if there is anything that I'm missing here.

Propose a Solution

Looking into the implementation in this SDK, it seems like a raw BaseChatModel implementation from scratch, only implementing the _generate() method. In the Python SDK, the standard Langchain ChatOpenAI class is used as a base and only the GenAI hub specific aspects are overwritten.

I am sure there is a good reason for this approach, however it means that any features (such as the upcoming native JSON Schema output from OpenAI) have to be manually implemented/added instead of relying on the existing Langchain implementation.

As described in the Langchain Docs https://js.langchain.com/docs/how_to/tool_calling/ Langchain Classes for LLMs that support tool calling should implement the .bindTools() method to parse langchain tools to the provider format (in this case OpenAI). The Langchain .withStructuredOutput() function also fails since .bindTools() is not implemented.

Describe Alternatives

It is possible to work around it by using other wrappers provided by Langchain, which use basic prompting to achieve a similar result. However, this is not as reliable, as noted by OpenAI in the release Blog for Structured Outputs: https://openai.com/index/introducing-structured-outputs-in-the-api/. This strict=true mode mentioned in the blog is also already supported by the native Langchain OpenAI implementation, however the models available in the GenAI hub currently don't support it anyways.

EDIT: The gpt-4o-mini model in GenAI hub now supports the json_schema with strict: true. This means it would be beneficial to both allow the use of function calling with strict: true as well as Structured Outputs using json_schema and strict: true. This could dramatically increase the reliability of GenAI Applications relying on a certain structured output format, as described in the aforementioned Blog by OpenAI.

Affected Development Phase

Development

Impact

Impaired

Timeline

No response

Additional Context

No response

[jjtang1985]

Thanks for raising up!
I've created a backlog item.

[krodyrobi]

I've also hit the same problem, any timeline available?

[ZhongpinWang]

Hi @alexbau01 , @krodyrobi ,

Updates:

• Both our LangChain clients support bindTools() now. Check the agent tutorial for usage example.
• Response format / withStructuredOutput() is still WIP. However, as documented by LangChain, you can use tool calling to return structured output.

[alexbau01]

Hi @ZhongpinWang, thanks picking this up! Bind tools looks great, nice example! This will make it much easier to build Agents, and awesome that it is also supported with the Langchain Orchestration Client.

Looking forward to the withStructuredOutput support. Good having the workaround with bindTools now, but it is a bit hacky so will be great to also have the dedicated method supported.

[deekshas8]

Hi @alexbau01,

withStructuredOutput() is now supported in the LangChain OpenAI client with v1.17.0.