Support for More LLms / configurable ExcecutableID

[MichaelSchmid-AGI]

Describe the Problem

Hi there,
i am struggling to get LLMS running that arent from OpenAI with your langchain module. Per default you seem to filter out LLMS which are in a different "excecutableId" than "azure-openai"

This makes using different Models seemingly impossible (for now)

Propose a Solution

I would suggest that you allow to pass a excecutableId when initializing a langchain chat client

const chatClient = new AzureOpenAiChatClient({
  modelName: 'meta--llama3.1-70b-instruct',
  excecutableID:'aicore-opensource'
});

this should allow the usage of other models quite easily without having to rewrite much.

The chat-completion api ( for example when using llama or mixtral) seems identical to payloads working with OpenAI models.

POST {baseurl}/v2/inference/deployments/{deploymentID}/chat/completions
Body for GPT 4o:

{
    "messages": [
        {
            "role": "user",
            "content": "test"
        }
    ],
    "model": "gpt-4o", 
    "max_tokens": 100,
    "temperature": 0.0,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "stop": "null"
}

Body For Llama 3.1 Instruct

{
  "messages": [
    {
      "role": "user",
      "content": "test"
    }
  ],
  "model": "meta--llama3.1-70b-instruct", 
  "max_tokens": 100,
  "temperature": 0.0,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "stop": "null"
}

Describe Alternatives

No response

Affected Development Phase

Development

Impact

Inconvenience

Timeline

No response

Additional Context

No response

[jjtang1985]

Hi @MichaelSchmid-AGI ,
Thanks for reaching out.

I guess the feature request is about supporting more direct LLM consumption.
I'm not sure whether we will reuse the same AzureOpenAiChatClient, though.

Are you aware our orchestration package: @sap-ai-sdk/orchestration and the orchestration service?
If you want to consume other LLMs, you can use the harmonised API offered by orchestration service now.

[skye0402]

@jjtang1985 the idea of using orchestration is good to have a unified entry point. However, I'm not clear how multimodal capabilities are provided by the orchestration. I tried to pass an image into it and it gave a strange error. I think it's just not working. Also it might add unwanted latency. Next, not sure how compatible it is with langchain? So far I need to rely on the Python SDK as it offers most models. Would be great to see Anthropic and Gemini supported soon.

[Mohan-Sharma]

@jjtang1985 the idea of using orchestration is good to have a unified entry point. However, I'm not clear how multimodal capabilities are provided by the orchestration. I tried to pass an image into it and it gave a strange error. I think it's just not working. Also it might add unwanted latency. Next, not sure how compatible it is with langchain? So far I need to rely on the Python SDK as it offers most models. Would be great to see Anthropic and Gemini supported soon.

Agreed, it’s crucial that the ai-sdk-js supports ChatVertexAI, ChatAnthropic, BedrockEmbeddings, and VertexAIEmbeddings. Even if we opt for Orchestration, the @sap-ai-sdk/orchestration module’s lack of support for LangChain adds unnecessary complexity, making it cumbersome to work with while leveraging the AI SDK.

[jjtang1985]

Hi @skye0402 ,

Thank you for reaching out!

I tried to pass an image into it and it gave a strange error.

Using image as input, as a feature has been supported by orchestration from last month.
We are also looking at it.

Also it might add unwanted latency.

As far as I know, the team did some tests and there are no major findings in terms of the latency.
Please open an issue for AI core, if you detect something.

Next, not sure how compatible it is with langchain?

So far, not supported.
Here is the feature request, please add your detailed requirements there, if possible.

[jjtang1985]

Hi @Mohan-Sharma ,

Even if we opt for Orchestration, the @sap-ai-sdk/orchestration module’s lack of support for LangChain adds unnecessary complexity, making it cumbersome to work with while leveraging the AI SDK.

Here is the #176, please add your detailed requirements there, if possible.

Thank you!

[AjitKP91]

@jjtang1985 Is there a timeline on supporting other LLMs directly instead of Orchestration? Currently there is a hug difference between AI SDK for JS and Python.

[andry900]

@jjtang1985 Is there a timeline on supporting other LLMs directly instead of Orchestration? Currently there is a hug difference between AI SDK for JS and Python.

Any update on this? We need to call the LLMs directly, without the Orchestration

[jjtang1985]

Hi @AjitKP91 and @andry900 ,

Thanks for reaching out.

A short answer would be no, we dont have timeline for that.

From SAP AI Core perspective, using orchestration should be considered as default strategy instead of direct LLM calls.
Could you please let us know the reason why you would not like to use the orchestration?

Previously, there are 2 major reasons:

• LangChain with orchestration is needed, which is not an issue any more.
• Embedding endpoint is needed, which will happen soon.

Best regards,
Junjie