diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md index c511468456b..8605f8f7eb2 100644 --- a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md @@ -1,238 +1,1263 @@ --- Description: "" -date: "2025-03-19" +date: "2025-12-02" lastmod: "" tags: [] -title: ChatModel - ARK +title: ChatModel - ark weight: 0 --- -## **Basic Introduction** +A Volcengine Ark model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. -Ark is an implementation of the ChatModel interface, used for interacting with the Volcano Engine Ark Runtime service. Ark Runtime is a LLM runtime service provided by Volcano Engine, offering a wide range of model options and complete API functionalities. This component interacts with the service through the Ark Runtime Go SDK, allowing the invocation of models deployed on the Volcano Engine, such as the Doubao LLM and the Shadow Moon LLM. This component implements the [Eino: ChatModel guide](/docs/eino/core_modules/components/chat_model_guide). +This package provides two distinct models: +- **ChatModel**: For text-based and multi-modal chat completions. +- **ImageGenerationModel**: For generating images from text prompts or image. +- **ResponseAPI**: Contains methods and other services that help with interacting with the openai API. -## **Usage** +## Features -### **Component Initialization** +- Implements `github.com/cloudwego/eino/components/model.Model` +- Easy integration with Eino's model system +- Configurable model parameters +- Support for both chat completion, image generation and response api +- Support for streaming responses +- Custom response parsing support +- Flexible model configuration -The Ark model is initialized through the `NewChatModel` function with the following key configuration parameters: +## Installation + +```bash +go get github.com/cloudwego/eino-ext/components/model/ark@latest +``` + +--- + +## Chat Completion + +This model is used for standard chat and text generation tasks. + +### Quick Start + +Here's a quick example of how to use the `ChatModel`: + +```go +package main + +import ( + "context" + "encoding/json" + "errors" + "io" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ark" +) + +func main() { + ctx := context.Background() + + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + inMsgs := []*schema.Message{ + { + Role: schema.User, + Content: "how do you generate answer for user question as a machine, please answer in short?", + }, + } + + msg, err := chatModel.Generate(ctx, inMsgs) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("generate output: \n") + log.Printf(" request_id: %s\n") + respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) + + sr, err := chatModel.Stream(ctx, inMsgs) + if err != nil { + log.Fatalf("Stream failed, err=%v", err) + } + + chunks := make([]*schema.Message, 0, 1024) + for { + msgChunk, err := sr.Recv() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + log.Fatalf("Stream Recv failed, err=%v", err) + } + + chunks = append(chunks, msgChunk) + } + + msg, err = schema.ConcatMessages(chunks) + if err != nil { + log.Fatalf("ConcatMessages failed, err=%v", err) + } + + log.Printf("stream final output: \n") + log.Printf(" request_id: %s\n") + respBody, _ = json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) +} +``` + +### Configuration + +The `ChatModel` can be configured using the `ark.ChatModelConfig` struct: + +```go +type ChatModelConfig struct { + // Timeout specifies the maximum duration to wait for API responses + // Optional. Default: 10 minutes + Timeout *time.Duration `json:"timeout"` + + // RetryTimes specifies the number of retry attempts for failed API calls + // Optional. Default: 2 + RetryTimes *int `json:"retry_times"` + + // BaseURL specifies the base URL for Ark service + // Optional. Default: "https://ark.cn-beijing.volces.com/api/v3" + BaseURL string `json:"base_url"` + // Region specifies the region where Ark service is located + // Optional. Default: "cn-beijing" + Region string `json:"region"` + + // The following three fields are about authentication - either APIKey or AccessKey/SecretKey pair is required + // For authentication details, see: https://www.volcengine.com/docs/82379/1298459 + // APIKey takes precedence if both are provided + APIKey string `json:"api_key"` + AccessKey string `json:"access_key"` + SecretKey string `json:"secret_key"` + + // The following fields correspond to Ark's chat completion API parameters + // Ref: https://www.volcengine.com/docs/82379/1298454 + + // Model specifies the ID of endpoint on ark platform + // Required + Model string `json:"model"` + + // MaxTokens limits the maximum number of tokens that can be generated in the chat completion and the range of values is [0, 4096] + // Optional. Default: 4096 + MaxTokens *int `json:"max_tokens,omitempty"` + + // Temperature specifies what sampling temperature to use + // Generally recommend altering this or TopP but not both + // Range: 0.0 to 1.0. Higher values make output more random + // Optional. Default: 1.0 + Temperature *float32 `json:"temperature,omitempty"` + + // TopP controls diversity via nucleus sampling + // Generally recommend altering this or Temperature but not both + // Range: 0.0 to 1.0. Lower values make output more focused + // Optional. Default: 0.7 + TopP *float32 `json:"top_p,omitempty"` + + // Stop sequences where the API will stop generating further tokens + // Optional. Example: []string{"\n", "User:"} + Stop []string `json:"stop,omitempty"` + + // FrequencyPenalty prevents repetition by penalizing tokens based on frequency + // Range: -2.0 to 2.0. Positive values decrease likelihood of repetition + // Optional. Default: 0 + FrequencyPenalty *float32 `json:"frequency_penalty,omitempty"` + + // LogitBias modifies likelihood of specific tokens appearing in completion + // Optional. Map token IDs to bias values from -100 to 100 + LogitBias map[string]int `json:"logit_bias,omitempty"` + + // PresencePenalty prevents repetition by penalizing tokens based on presence + // Range: -2.0 to 2.0. Positive values increase likelihood of new topics + // Optional. Default: 0 + PresencePenalty *float32 `json:"presence_penalty,omitempty"` + + // CustomHeader the http header passed to model when requesting model + CustomHeader map[string]string `json:"custom_header"` +} +``` + +### Request Options + +The `ChatModel` supports various request options to customize the behavior of API calls. Here are the available options: ```go -import "github.com/cloudwego/eino-ext/components/model/ark" - -model, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ - // Service Configuration - BaseURL: "https://ark.cn-beijing.volces.com/api/v3", // Service Address - Region: "cn-beijing", // Region - HTTPClient: httpClient, // Custom HTTP Client - Timeout: &timeout, // Timeout Duration - RetryTimes: &retries, // Retry Attempts - - // Authentication Configuration (Choose one of the two) - APIKey: "your-api-key", // API Key Authentication - AccessKey: "your-ak", // AK/SK Authentication - SecretKey: "your-sk", - - // Model Configuration - Model: "endpoint-id", // Model Endpoint ID - - // Generation Parameters - MaxTokens: &maxTokens, // Maximum Generation Length - Temperature: &temp, // Temperature - TopP: &topP, // Top-P Sampling - Stop: []string{}, // Stop Words - FrequencyPenalty: &fp, // Frequency Penalty - PresencePenalty: &pp, // Presence Penalty - RepetitionPenalty: &rp, // Repetition Penalty - N: &n, // Number of Generations - - // Advanced Parameters - ResponseFormat: &format, // Response Format - LogitBias: map[string]int{}, // Token Bias - LogProbs: &logProbs, // Return Log Probabilities - TopLogProbs: &topLp, // Number of Top K Probabilities - User: &user, // User Identifier -}) +// WithCustomHeader sets custom headers for a single request +// the headers will override all the headers given in ChatModelConfig.CustomHeader +func WithCustomHeader(m map[string]string) model.Option {} ``` -### **Generating Conversations** +--- + +## Image Generation -Conversation generation supports both normal mode and streaming mode: +This model is used specifically for generating images from text prompts. + +### Quick Start + +Here's a quick example of how to use the `ImageGenerationModel`: ```go +package main + +import ( + "context" + "encoding/json" + "log" + "os" + + "github.com/cloudwego/eino/schema" + "github.com/cloudwego/eino-ext/components/model/ark" +) + func main() { - // Normal Mode - response, err := model.Generate(ctx, messages) + ctx := context.Background() + + // Get ARK_API_KEY and an image generation model ID + imageGenerationModel, err := ark.NewImageGenerationModel(ctx, &ark.ImageGenerationConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_IMAGE_MODEL_ID"), // Use an appropriate image model ID + }) + + if err != nil { + log.Fatalf("NewImageGenerationModel failed, err=%v", err) + } + + inMsgs := []*schema.Message{ + { + Role: schema.User, + Content: "a photo of a cat sitting on a table", + }, + } + + msg, err := imageGenerationModel.Generate(ctx, inMsgs) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("\ngenerate output: \n") + respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) + + sr, err := imageGenerationModel.Stream(ctx, inMsgs) + if err != nil { + log.Fatalf("Stream failed, err=%v", err) + } + + log.Printf("stream output: \n") + index := 0 + for { + msgChunk, err := sr.Recv() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + log.Fatalf("Stream Recv failed, err=%v", err) + } + + respBody, _ = json.MarshalIndent(msgChunk, " ", " ") + log.Printf("stream chunk %d: body: %s\n", index, string(respBody)) + index++ + } +} +``` + +### Configuration + +The `ImageGenerationModel` can be configured using the `ark.ImageGenerationConfig` struct: + +```go + +type ImageGenerationConfig struct { + // For authentication, APIKey is required as the image generation API only supports API Key authentication. + // For authentication details, see: https://www.volcengine.com/docs/82379/1298459 + // Required + APIKey string `json:"api_key"` + + // Model specifies the ID of endpoint on ark platform + // Required + Model string `json:"model"` + + // Timeout specifies the maximum duration to wait for API responses + // If HTTPClient is set, Timeout will not be used. + // Optional. Default: 10 minutes + Timeout *time.Duration `json:"timeout"` + + // HTTPClient specifies the client to send HTTP requests. + // If HTTPClient is set, Timeout will not be used. + // Optional. Default &http.Client{Timeout: Timeout} + HTTPClient *http.Client `json:"http_client"` - // Streaming Mode - stream, err := model.Stream(ctx, messages) + // RetryTimes specifies the number of retry attempts for failed API calls + // Optional. Default: 2 + RetryTimes *int `json:"retry_times"` + + // BaseURL specifies the base URL for Ark service + // Optional. Default: "https://ark.cn-beijing.volces.com/api/v3" + BaseURL string `json:"base_url"` + + // Region specifies the region where Ark service is located + // Optional. Default: "cn-beijing" + Region string `json:"region"` + + // The following fields correspond to Ark's image generation API parameters + // Ref: https://www.volcengine.com/docs/82379/1541523 + + // Size specifies the dimensions of the generated image. + // It can be a resolution keyword (e.g., "1K", "2K", "4K") or a custom resolution + // in "{width}x{height}" format (e.g., "1920x1080"). + // When using custom resolutions, the total pixels must be between 1280x720 and 4096x4096, + // and the aspect ratio (width/height) must be between 1/16 and 16. + // Optional. Defaults to "2048x2048". + Size string `json:"size"` + + // SequentialImageGeneration determines if the model should generate a sequence of images. + // Possible values: + // - "auto": The model decides whether to generate multiple images based on the prompt. + // - "disabled": Only a single image is generated. + // Optional. Defaults to "disabled". + SequentialImageGeneration SequentialImageGeneration `json:"sequential_image_generation"` + + // SequentialImageGenerationOption sets the maximum number of images to generate when + // SequentialImageGeneration is set to "auto". + // The value must be between 1 and 15. + // Optional. Defaults to 15. + SequentialImageGenerationOption *model.SequentialImageGenerationOptions `json:"sequential_image_generation_option"` + + // ResponseFormat specifies how the generated image data is returned. + // Possible values: + // - "url": A temporary URL to download the image (valid for 24 hours). + // - "b64_json": The image data encoded as a Base64 string in the response. + // Optional. Defaults to "url". + ResponseFormat ImageResponseFormat `json:"response_format"` + + // DisableWatermark, if set to true, removes the "AI Generated" watermark + // from the bottom-right corner of the image. + // Optional. Defaults to false. + DisableWatermark bool `json:"disable_watermark"` +} +``` + + +## examples + +### generate + +```go + +package main + +import ( + "context" + "encoding/json" + "errors" + "io" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ark" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + inMsgs := []*schema.Message{ + { + Role: schema.User, + Content: "how do you generate answer for user question as a machine, please answer in short?", + }, + } + + msg, err := chatModel.Generate(ctx, inMsgs) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("\ngenerate output: \n") + log.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) + + sr, err := chatModel.Stream(ctx, inMsgs) + if err != nil { + log.Fatalf("Stream failed, err=%v", err) + } + + chunks := make([]*schema.Message, 0, 1024) + for { + msgChunk, err := sr.Recv() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + log.Fatalf("Stream Recv failed, err=%v", err) + } + + chunks = append(chunks, msgChunk) + } + + msg, err = schema.ConcatMessages(chunks) + if err != nil { + log.Fatalf("ConcatMessages failed, err=%v", err) + } + + log.Printf("stream final output: \n") + log.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + respBody, _ = json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) } + ``` -Sample message format: +### generate_with_image + +```go + +package main + +import ( + "context" + "encoding/base64" + "log" + "os" + + "github.com/cloudwego/eino/components/prompt" + "github.com/cloudwego/eino/compose" + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ark" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + multiModalMsg := schema.UserMessage("") + image, err := os.ReadFile("./examples/generate_with_image/eino.png") + if err != nil { + log.Fatalf("os.ReadFile failed, err=%v\n", err) + } + + imageStr := base64.StdEncoding.EncodeToString(image) + + multiModalMsg.UserInputMultiContent = []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "What do you see in this image?", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: &imageStr, + MIMEType: "image/png", + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + multiModalMsg, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("Ark ChatModel output: \n%v", resp) + + // demonstrate how to use ChatTemplate to generate with image + imgPlaceholder := "{img}" + ctx = context.Background() + chain := compose.NewChain[map[string]any, *schema.Message]() + _ = chain.AppendChatTemplate(prompt.FromMessages(schema.FString, + &schema.Message{ + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "What do you see in this image?", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: &imgPlaceholder, + MIMEType: "image/png", + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + }, + })) + _ = chain.AppendChatModel(chatModel) + r, err := chain.Compile(ctx) + if err != nil { + log.Fatalf("Compile failed, err=%v", err) + } + + resp, err = r.Invoke(ctx, map[string]any{ + "img": imageStr, + }) + if err != nil { + log.Fatalf("Run failed, err=%v", err) + } + + log.Printf("Ark ChatModel output with ChatTemplate: \n%v", resp) +} + +``` + +### stream + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/ark" + "github.com/cloudwego/eino/schema" + arkModel "github.com/volcengine/volcengine-go-sdk/service/arkruntime/model" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v", err) + return + } -> Note: Whether images in multimodal are supported depends on the specific model + streamMsgs, err := chatModel.Stream(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "as a machine, how do you answer user's question?", + }, + }, ark.WithReasoningEffort(arkModel.ReasoningEffortHigh)) + + if err != nil { + log.Printf("Generate failed, err=%v", err) + return + } + + defer streamMsgs.Close() // do not forget to close the stream + + msgs := make([]*schema.Message, 0) + + log.Printf("typewriter output:") + for { + msg, err := streamMsgs.Recv() + if err == io.EOF { + break + } + msgs = append(msgs, msg) + if err != nil { + log.Printf("\nstream.Recv failed, err=%v", err) + return + } + fmt.Print(msg.Content) + } + + msg, err := schema.ConcatMessages(msgs) + if err != nil { + log.Printf("ConcatMessages failed, err=%v", err) + return + } + + log.Printf("output: %s\n", msg.Content) +} + +``` + +### intent_tool ```go + +package main + +import ( + "context" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/ark" + "github.com/cloudwego/eino/schema" +) + func main() { - messages := []*schema.Message{ - // System Message - schema.SystemMessage("You are an assistant"), - - // Multimodal Message (including images) - { - Role: schema.User, - MultiContent: []schema.ChatMessagePart{ - { - Type: schema.ChatMessagePartTypeText, - Text: "What is in this picture?", - }, - { - Type: schema.ChatMessagePartTypeImageURL, - ImageURL: &schema.ChatMessageImageURL{ - URL: "https://example.com/image.jpg", - Detail: schema.ImageURLDetailAuto, - }, - }, - }, - }, - } + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v", err) + return + } + + err = chatModel.BindTools([]*schema.ToolInfo{ + { + Name: "user_company", + Desc: "Query the user's company and position information based on their name and email", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "The user's name", + }, + "email": { + Type: "string", + Desc: "The user's email", + }, + }), + }, + { + Name: "user_salary", + Desc: "Query the user's salary information based on their name and email", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "The user's name", + }, + "email": { + Type: "string", + Desc: "The user's email", + }, + }), + }, + }) + if err != nil { + log.Printf("BindForcedTools failed, err=%v", err) + return + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "You are a real estate agent. Use the user_company and user_salary APIs to provide relevant property information based on the user's salary and job. Email is required", + }, + { + Role: schema.User, + Content: "My name is zhangsan, and my email is zhangsan@bytedance.com. Please recommend some suitable houses for me.", + }, + }) + + if err != nil { + log.Printf("Generate failed, err=%v", err) + return + } + + log.Printf("output: \n%v", resp) } + ``` -### **Tool Invocation** +### image_generate + +```go + +package main + +import ( + "context" + "encoding/json" + "errors" + "io" + "log" + "os" + + "github.com/volcengine/volcengine-go-sdk/service/arkruntime/model" + + "github.com/cloudwego/eino-ext/components/model/ark" + "github.com/cloudwego/eino/schema" +) + +func ptr[T any](v T) *T { + return &v +} + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + imageGenerationModel, err := ark.NewImageGenerationModel(ctx, &ark.ImageGenerationConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + + // Control the size of image generated by the model. + Size: "1K", + + // Control whether to generate a set of images. + SequentialImageGeneration: ark.SequentialImageGenerationAuto, + + // Control the maximum number of images to generate + SequentialImageGenerationOption: &model.SequentialImageGenerationOptions{ + MaxImages: ptr(2), + }, + + // Control the format of the generated jpeg image. + ResponseFormat: ark.ImageResponseFormatURL, + + // Control whether to add a watermark to the generated image + DisableWatermark: false, + }) + + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + inMsgs := []*schema.Message{ + { + Role: schema.User, + Content: "generate two images of a cat", + }, + } -Supported tools binding: + // Use ImageGeneration API + msg, err := imageGenerationModel.Generate(ctx, inMsgs) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("\ngenerate output: \n") + respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) + + sr, err := imageGenerationModel.Stream(ctx, inMsgs) + if err != nil { + log.Fatalf("Stream failed, err=%v", err) + } + + log.Printf("stream output: \n") + index := 0 + chunks := make([]*schema.Message, 0, 1024) + for { + msgChunk, err := sr.Recv() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + log.Fatalf("Stream Recv failed, err=%v", err) + } + + chunks = append(chunks, msgChunk) + + respBody, _ = json.MarshalIndent(msgChunk, " ", " ") + log.Printf("stream chunk %d: body: %s\n", index, string(respBody)) + index++ + } + + msg, err = schema.ConcatMessages(chunks) + if err != nil { + log.Fatalf("ConcatMessages failed, err=%v", err) + } + log.Printf("stream final output: \n") + log.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + respBody, _ = json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) +} + +``` + +### prefixcache-contextapi ```go -// Define tool -tools := []*schema.ToolInfo{ - { - Name: "search", - Desc: "Search information", - ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{ - "query": { - Type: schema.String, - Desc: "Search keyword", - Required: true, - }, - }), - }, + +package main + +import ( + "context" + "encoding/json" + "io" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ark" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + info, err := chatModel.CreatePrefixCache(ctx, []*schema.Message{ + schema.UserMessage("my name is megumin"), + }, 3600) + if err != nil { + log.Fatalf("CreatePrefix failed, err=%v", err) + } + + inMsgs := []*schema.Message{ + { + Role: schema.User, + Content: "what is my name?", + }, + } + + msg, err := chatModel.Generate(ctx, inMsgs, ark.WithCache(&ark.CacheOption{ + APIType: ark.ContextAPI, + ContextID: &info.ContextID, + })) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("\ngenerate output: \n") + log.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) + + outStreamReader, err := chatModel.Stream(ctx, inMsgs, ark.WithCache(&ark.CacheOption{ + APIType: ark.ContextAPI, + ContextID: &info.ContextID, + })) + if err != nil { + log.Fatalf("Stream failed, err=%v", err) + } + + var msgs []*schema.Message + for { + item, e := outStreamReader.Recv() + if e == io.EOF { + break + } + if e != nil { + log.Fatal(e) + } + + msgs = append(msgs, item) + } + msg, err = schema.ConcatMessages(msgs) + if err != nil { + log.Fatalf("ConcatMessages failed, err=%v", err) + } + log.Printf("\nstream output: \n") + log.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + respBody, _ = json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) } -// Bind tools -err := model.BindTools(tools) ``` -> For information related to tools, please refer to [Eino: ToolsNode guide](/docs/eino/core_modules/components/tools_node_guide) +### prefixcache-responsesapi + +```go + +package main -### **Complete Usage Example** +import ( + "context" + "encoding/json" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ark" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + Cache: &ark.CacheConfig{ + APIType: ptrOf(ark.ResponsesAPI), + }, + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + err = chatModel.BindTools([]*schema.ToolInfo{ + { + Name: "article_content_extractor", + Desc: "Extract key statements and chapter summaries from the provided article content", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "content": { + Type: schema.String, + Desc: "The full article content to analyze and extract key information from", + Required: true, + }, + }), + }, + }) + + if err != nil { + log.Fatalf("BindTools failed, err=%v", err) + } + + // create response prefix cache, note: more than 1024 tokens are required, otherwise the prefix cache cannot be created + cacheInfo, err := chatModel.CreatePrefixCache(ctx, []*schema.Message{ + schema.SystemMessage(`Once upon a time, in a quaint little village surrounded by vast green forests and blooming meadows, there lived a spirited young girl known as Little Red Riding Hood. She earned her name from the vibrant red cape that her beloved grandmother had sewn for her, a gift that she cherished deeply. This cape was more than just a piece of clothing; it was a symbol of the bond between her and her grandmother, who lived on the other side of the great woods, near a sparkling brook that bubbled merrily all year round. + + One sunny morning, Little Red Riding Hood's mother called her into the cozy kitchen, where the aroma of freshly baked bread filled the air. “My dear,” she said, “your grandmother isn’t feeling well today. I want you to take her this basket of treats. There are some delicious cakes, a jar of honey, and her favorite herbal tea. Can you do that for me?” + + Little Red Riding Hood’s eyes sparkled with excitement as she nodded eagerly. “Yes, Mama! I’ll take good care of them!” Her mother handed her a beautifully woven basket, filled to the brim with goodies, and reminded her, “Remember to stay on the path and don’t talk to strangers.” + + “I promise, Mama!” she replied confidently, pulling her red hood over her head and setting off on her adventure. The sun shone brightly, and birds chirped merrily as she walked, making her feel like she was in a fairy tale. + + As she journeyed through the woods, the tall trees whispered secrets to one another, and colorful flowers danced in the gentle breeze. Little Red Riding Hood was so enchanted by the beauty around her that she began to hum a tune, her voice harmonizing with the sounds of nature. + + However, unbeknownst to her, lurking in the shadows was a cunning wolf. The wolf was known throughout the forest for his deceptive wit and insatiable hunger. He watched Little Red Riding Hood with keen interest, contemplating his next meal. + + “Good day, little girl!” the wolf called out, stepping onto the path with a friendly yet sly smile. + + Startled, she halted and took a step back. “Hello there! I’m just on my way to visit my grandmother,” she replied, clutching the basket tightly. + + “Ah, your grandmother! I know her well,” the wolf said, his eyes glinting with mischief. “Why don’t you pick some lovely flowers for her? I’m sure she would love them, and I’m sure there are many beautiful ones just off the path.” + + Little Red Riding Hood hesitated for a moment but was easily convinced by the wolf’s charming suggestion. “That’s a wonderful idea! Thank you!” she exclaimed, letting her curiosity pull her away from the safety of the path. As she wandered deeper into the woods, her gaze fixed on the vibrant blooms, the wolf took a shortcut towards her grandmother’s house. + + When the wolf arrived at Grandma’s quaint cottage, he knocked on the door with a confident swagger. “It’s me, Little Red Riding Hood!” he shouted in a high-pitched voice to mimic the girl. + + “Come in, dear!” came the frail voice of the grandmother, who had been resting on her cozy bed, wrapped in warm blankets. The wolf burst through the door, his eyes gleaming with the thrill of his plan. + + With astonishing speed, the wolf gulped down the unsuspecting grandmother whole. Afterward, he dressed in her nightgown, donning her nightcap and climbing into her bed. He lay there, waiting for Little Red Riding Hood to arrive, concealing his wicked smile behind a facade of innocence. + + Meanwhile, Little Red Riding Hood was merrily picking flowers, completely unaware of the impending danger. After gathering a beautiful bouquet of wildflowers, she finally made her way back to the path and excitedly skipped towards her grandmother’s cottage. + + Upon arriving, she noticed the door was slightly ajar. “Grandmother, it’s me!” she called out, entering the dimly lit home. It was silent, with only the faint sound of an old clock ticking in the background. She stepped into the small living room, a feeling of unease creeping over her. + + “Grandmother, are you here?” she asked, peeking into the bedroom. There, she saw a figure lying under the covers. + + “Grandmother, what big ears you have!” she exclaimed, taking a few cautious steps closer. + + “All the better to hear you with, my dear,” the wolf replied in a voice that was deceptively sweet. + + “Grandmother, what big eyes you have!” Little Red Riding Hood continued, now feeling an unsettling chill in the air. + + “All the better to see you with, my dear,” the wolf said, his eyes narrowing as he tried to contain his glee. + + “Grandmother, what big teeth you have!” she exclaimed, the terror flooding her senses as she began to realize this was no ordinary visit. + + “All the better to eat you with!” the wolf roared, springing out of the bed with startling speed. + + Just as the wolf lunged towards her, a brave woodsman, who had been passing by the cottage and heard the commotion, burst through the door. His strong presence was a beacon of hope in the dire situation. “Stay back, wolf!” he shouted with authority, brandishing his axe. + + The wolf, taken aback by the sudden intrusion, hesitated for a moment. Before he could react, the woodsman swung his axe with determination, and with a swift motion, he drove the wolf away, rescuing Little Red Riding Hood and her grandmother from certain doom. + + Little Red Riding Hood was shaking with fright, but relief washed over her as the woodsman helped her grandmother out from behind the bed where the wolf had hidden her. The grandmother, though shaken, was immensely grateful to the woodsman for his bravery. “Thank you so much! You saved us!” she cried, embracing him warmly. + + Little Red Riding Hood, still in shock but filled with gratitude, looked up at the woodsman and said, “I promise I will never stray from the path again. Thank you for being our hero!” + + From that day on, the woodland creatures spoke of the brave woodsman who saved Little Red Riding Hood and her grandmother. Little Red Riding Hood learned a valuable lesson about being cautious and listening to her mother’s advice. The bond between her and her grandmother grew stronger, and they often reminisced about that day’s adventure over cups of tea, surrounded by cookies and laughter. + + To ensure safety, Little Red Riding Hood always took extra precautions when traveling through the woods, carrying a small whistle her grandmother had given her. It would alert anyone nearby if she ever found herself in trouble again. + + And so, in the heart of that small village, life continued, filled with love, laughter, and the occasional adventure, as Little Red Riding Hood and her grandmother thrived, forever grateful for the friendship of the woodsman who had acted as their guardian that fateful day. + + And they all lived happily ever after. + + The end.`), + }, 300) + if err != nil { + log.Fatalf("CreatePrefixCache failed, err=%v", err) + } + + // use cache information in subsequent requests + cacheOpt := &ark.CacheOption{ + APIType: ark.ResponsesAPI, + HeadPreviousResponseID: &cacheInfo.ResponseID, + } + + outMsg, err := chatModel.Generate(ctx, []*schema.Message{ + schema.UserMessage("What is the main idea expressed above？"), + }, ark.WithCache(cacheOpt)) + + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + respID, ok := ark.GetResponseID(outMsg) + if !ok { + log.Fatalf("not found response id in message") + } + + log.Printf("\ngenerate output: \n") + log.Printf(" request_id: %s\n", respID) + respBody, _ := json.MarshalIndent(outMsg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) +} +func ptrOf[T any](v T) *T { + return &v + +} + +``` -#### **Direct Conversation** +### sessioncache-contextapi ```go + package main import ( - "context" - "time" - - "github.com/cloudwego/eino-ext/components/model/ark" - "github.com/cloudwego/eino/schema" + "context" + "encoding/json" + "fmt" + "io" + "log" + "os" + "time" + + arkModel "github.com/volcengine/volcengine-go-sdk/service/arkruntime/model" + + "github.com/cloudwego/eino-ext/components/model/ark" + "github.com/cloudwego/eino/schema" ) func main() { - ctx := context.Background() - - timeout := 30 * time.Second - // Initialize model - model, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ - APIKey: "your-api-key", - Region: "cn-beijing", - Model: "endpoint-id", - Timeout: &timeout, - }) - if err != nil { - panic(err) - } - - // Prepare messages - messages := []*schema.Message{ - schema.SystemMessage("You are an assistant"), - schema.UserMessage("Introduce Volcano Engine"), - } - - // Generate response - response, err := model.Generate(ctx, messages) - if err != nil { - panic(err) - } - - // Handle response - println(response.Content) - - // Get token usage information - if usage := response.ResponseMeta.Usage; usage != nil { - println("Prompt Tokens:", usage.PromptTokens) - println("Completion Tokens:", usage.CompletionTokens) - println("Total Tokens:", usage.TotalTokens) - } + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + instructions := []*schema.Message{ + schema.SystemMessage("Your name is superman"), + } + + cacheInfo, err := chatModel.CreateSessionCache(ctx, instructions, 86400, nil) + if err != nil { + log.Fatalf("CreateSessionCache failed, err=%v", err) + } + + thinking := &arkModel.Thinking{ + Type: arkModel.ThinkingTypeDisabled, + } + + cacheOpt := &ark.CacheOption{ + APIType: ark.ContextAPI, + ContextID: &cacheInfo.ContextID, + SessionCache: &ark.SessionCacheConfig{ + EnableCache: true, + TTL: 86400, + }, + } + + msg, err := chatModel.Generate(ctx, instructions, + ark.WithThinking(thinking), + ark.WithCache(cacheOpt)) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + <-time.After(500 * time.Millisecond) + + msg, err = chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "What's your name?", + }, + }, + ark.WithThinking(thinking), + ark.WithCache(cacheOpt)) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + fmt.Printf("\ngenerate output: \n") + fmt.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + respBody, _ := json.MarshalIndent(msg, " ", " ") + fmt.Printf(" body: %s\n", string(respBody)) + + outStreamReader, err := chatModel.Stream(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "What do I ask you last time?", + }, + }, + ark.WithThinking(thinking), + ark.WithCache(cacheOpt)) + if err != nil { + log.Fatalf("Stream failed, err=%v", err) + } + + fmt.Println("\ntypewriter output:") + var msgs []*schema.Message + for { + item, e := outStreamReader.Recv() + if e == io.EOF { + break + } + if e != nil { + log.Fatal(e) + } + + fmt.Print(item.Content) + msgs = append(msgs, item) + } + + msg, err = schema.ConcatMessages(msgs) + if err != nil { + log.Fatalf("ConcatMessages failed, err=%v", err) + } + fmt.Print("\n\nstream output: \n") + fmt.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + respBody, _ = json.MarshalIndent(msg, " ", " ") + fmt.Printf(" body: %s\n", string(respBody)) } + ``` -#### **Streaming Conversation** +### sessioncache-responsesapi ```go + package main import ( - "context" - "time" - - "github.com/cloudwego/eino-ext/components/model/ark" - "github.com/cloudwego/eino/schema" + "context" + "fmt" + "io" + "log" + "os" + + arkModel "github.com/volcengine/volcengine-go-sdk/service/arkruntime/model" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ark" ) func main() { - ctx := context.Background() - - // Initialize model - model, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ - APIKey: "your-api-key", - Model: "ep-xxx", - }) - if err != nil { - panic(err) - } - - // Prepare messages - messages := []*schema.Message{ - schema.SystemMessage("You are an assistant"), - schema.UserMessage("Introduce Eino"), - } - - // Get streaming response - reader, err := model.Stream(ctx, messages) - if err != nil { - panic(err) - } - defer reader.Close() // Be sure to close - - // Handle streaming content - for { - chunk, err := reader.Recv() - if err != nil { - break - } - print(chunk.Content) - } + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + Cache: &ark.CacheConfig{ + SessionCache: &ark.SessionCacheConfig{ + EnableCache: true, + TTL: 86400, + }, + }, + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + thinking := &arkModel.Thinking{ + Type: arkModel.ThinkingTypeDisabled, + } + cacheOpt := &ark.CacheOption{ + APIType: ark.ResponsesAPI, + SessionCache: &ark.SessionCacheConfig{ + EnableCache: true, + TTL: 86400, + }, + } + + useMsgs := []*schema.Message{ + schema.UserMessage("Your name is superman"), + schema.UserMessage("What's your name?"), + schema.UserMessage("What do I ask you last time?"), + } + + var input []*schema.Message + for _, msg := range useMsgs { + input = append(input, msg) + + streamResp, err := chatModel.Stream(ctx, input, + ark.WithThinking(thinking), + ark.WithCache(cacheOpt)) + if err != nil { + log.Fatalf("Stream failed, err=%v", err) + } + + var messages []*schema.Message + for { + chunk, err := streamResp.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Fatalf("Recv of streamResp failed, err=%v", err) + } + messages = append(messages, chunk) + } + + resp, err := schema.ConcatMessages(messages) + if err != nil { + log.Fatalf("ConcatMessages of ark failed, err=%v", err) + } + + fmt.Printf("stream output: \n%v\n\n", resp) + + input = append(input, resp) + } } + ``` -## **Related Documentation** -- [Eino: ChatModel guide](/docs/eino/core_modules/components/chat_model_guide) -- [Eino: ToolsNode guide](/docs/eino/core_modules/components/tools_node_guide) -- [ChatModel - OpenAI](/docs/eino/ecosystem_integration/chat_model/chat_model_openai) -- [ChatModel - Ollama](/docs/eino/ecosystem_integration/chat_model/chat_model_ollama) -- [Volcano Engine Official Website](https://www.volcengine.com/product/doubao) + +## For More Details + +- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) +- [Volcengine Ark Model Documentation](https://www.volcengine.com/docs/82379/1263272) diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_arkbot.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_arkbot.md new file mode 100644 index 00000000000..f1610434a2d --- /dev/null +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_arkbot.md @@ -0,0 +1,339 @@ +--- +Description: "" +date: "2025-12-02" +lastmod: "" +tags: [] +title: ChatModel - arkbot +weight: 0 +--- + +A Volcengine Ark Bot implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. + +## Features + +- Implements `github.com/cloudwego/eino/components/model.ToolCallingChatModel` +- Easy integration with Eino's model system +- Configurable model parameters +- Support for chat completion +- Support for streaming responses +- Custom response parsing support +- Flexible model configuration + +## Installation + +```bash +go get github.com/cloudwego/eino-ext/components/model/arkbot@latest +``` + +## Quick Start + +Here's a quick example of how to use the Ark Bot: + +```go + +package main + +import ( + "context" + "encoding/json" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/arkbot" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := arkbot.NewChatModel(ctx, &arkbot.Config{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + inMsgs := []*schema.Message{ + { + Role: schema.User, + Content: "What's the weather in Beijing?", + }, + } + + msg, err := chatModel.Generate(ctx, inMsgs) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("generate output: \n") + log.Printf(" request_id: %s\n", arkbot.GetArkRequestID(msg)) + if bu, ok := arkbot.GetBotUsage(msg); ok { + bbu, _ := json.Marshal(bu) + log.Printf(" bot_usage: %s\n", string(bbu)) + } + if ref, ok := arkbot.GetBotChatResultReference(msg); ok { + bRef, _ := json.Marshal(ref) + log.Printf(" bot_chat_result_reference: %s\n", bRef) + } + respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) +} + + +``` + +## Configuration + +The model can be configured using the `arkbot.Config` struct: + +```go + + type Config struct { + // Timeout specifies the maximum duration to wait for API responses + // If HTTPClient is set, Timeout will not be used. + // Optional. Default: 10 minutes + Timeout *time.Duration `json:"timeout"` + + // HTTPClient specifies the client to send HTTP requests. + // If HTTPClient is set, Timeout will not be used. + // Optional. Default &http.Client{Timeout: Timeout} + HTTPClient *http.Client `json:"http_client"` + + // RetryTimes specifies the number of retry attempts for failed API calls + // Optional. Default: 2 + RetryTimes *int `json:"retry_times"` + + // BaseURL specifies the base URL for Ark service + // Optional. Default: "https://ark.cn-beijing.volces.com/api/v3" + BaseURL string `json:"base_url"` + // Region specifies the region where Ark service is located + // Optional. Default: "cn-beijing" + Region string `json:"region"` + + // The following three fields are about authentication - either APIKey or AccessKey/SecretKey pair is required + // For authentication details, see: https://www.volcengine.com/docs/82379/1298459 + // APIKey takes precedence if both are provided + APIKey string `json:"api_key"` + AccessKey string `json:"access_key"` + SecretKey string `json:"secret_key"` + + // The following fields correspond to Ark's chat completion API parameters + // Ref: https://www.volcengine.com/docs/82379/1298454 + + // Model specifies the ID of endpoint on ark platform + // Required + Model string `json:"model"` + + // MaxTokens limits the maximum number of tokens that can be generated in the chat completion. + // Optional. Default: 4096 + MaxTokens *int `json:"max_tokens,omitempty"` + + // Temperature specifies what sampling temperature to use + // Generally recommend altering this or TopP but not both + // Range: 0.0 to 1.0. Higher values make output more random + // Optional. Default: 1.0 + Temperature *float32 `json:"temperature,omitempty"` + + // TopP controls diversity via nucleus sampling + // Generally recommend altering this or Temperature but not both + // Range: 0.0 to 1.0. Lower values make output more focused + // Optional. Default: 0.7 + TopP *float32 `json:"top_p,omitempty"` + + // Stop sequences where the API will stop generating further tokens + // Optional. Example: []string{"\n", "User:"} + Stop []string `json:"stop,omitempty"` + + // FrequencyPenalty prevents repetition by penalizing tokens based on frequency + // Range: -2.0 to 2.0. Positive values decrease likelihood of repetition + // Optional. Default: 0 + FrequencyPenalty *float32 `json:"frequency_penalty,omitempty"` + + // LogitBias modifies likelihood of specific tokens appearing in completion + // Optional. Map token IDs to bias values from -100 to 100 + LogitBias map[string]int `json:"logit_bias,omitempty"` + + // PresencePenalty prevents repetition by penalizing tokens based on presence + // Range: -2.0 to 2.0. Positive values increase likelihood of new topics + // Optional. Default: 0 + PresencePenalty *float32 `json:"presence_penalty,omitempty"` + + // CustomHeader the http header passed to model when requesting model + CustomHeader map[string]string `json:"custom_header"` + + // LogProbs specifies whether to return log probabilities of the output tokens. + LogProbs bool `json:"log_probs"` + + // TopLogProbs specifies the number of most likely tokens to return at each token position, each with an associated log probability. + TopLogProbs int `json:"top_log_probs"` + + // ResponseFormat specifies the format that the model must output. + ResponseFormat *ResponseFormat `json:"response_format,omitempty"` + } +``` + +## Request Options + +The Ark model supports various request options to customize the behavior of API calls. Here are the available options: + +```go +// WithCustomHeader sets custom headers for a single request +// the headers will override all the headers given in ChatModelConfig.CustomHeader +func WithCustomHeader(m map[string]string) model.Option {} +``` + + +## examples + +### generate + +```go + +package main + +import ( + "context" + "encoding/json" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/arkbot" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := arkbot.NewChatModel(ctx, &arkbot.Config{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + inMsgs := []*schema.Message{ + { + Role: schema.User, + Content: "What's the weather in Beijing?", + }, + } + + msg, err := chatModel.Generate(ctx, inMsgs) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("generate output: \n") + log.Printf(" request_id: %s\n", arkbot.GetArkRequestID(msg)) + if bu, ok := arkbot.GetBotUsage(msg); ok { + bbu, _ := json.Marshal(bu) + log.Printf(" bot_usage: %s\n", string(bbu)) + } + if ref, ok := arkbot.GetBotChatResultReference(msg); ok { + bRef, _ := json.Marshal(ref) + log.Printf(" bot_chat_result_reference: %s\n", bRef) + } + respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) +} + +``` + +### stream + +```go + +package main + +import ( + "context" + "encoding/json" + "fmt" + "io" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/arkbot" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := arkbot.NewChatModel(ctx, &arkbot.Config{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v", err) + return + } + + streamMsgs, err := chatModel.Stream(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "What's the weather in Beijing?", + }, + }) + + if err != nil { + log.Printf("Generate failed, err=%v", err) + return + } + + defer streamMsgs.Close() // do not forget to close the stream + + msgs := make([]*schema.Message, 0) + + log.Printf("stream output:") + for { + msg, err := streamMsgs.Recv() + if err == io.EOF { + break + } + msgs = append(msgs, msg) + if err != nil { + log.Printf("\nstream.Recv failed, err=%v", err) + return + } + fmt.Print(msg.Content) + } + + msg, err := schema.ConcatMessages(msgs) + if err != nil { + log.Printf("ConcatMessages failed, err=%v", err) + return + } + + log.Printf("generate output: \n") + log.Printf(" request_id: %s\n", arkbot.GetArkRequestID(msg)) + if bu, ok := arkbot.GetBotUsage(msg); ok { + bbu, _ := json.Marshal(bu) + log.Printf(" bot_usage: %s\n", string(bbu)) + } + if ref, ok := arkbot.GetBotChatResultReference(msg); ok { + bRef, _ := json.Marshal(ref) + log.Printf(" bot_chat_result_reference: %s\n", bRef) + } + respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s\n", string(respBody)) +} + +``` + + + +## For More Details + +- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) +- [Volcengine Ark Model Documentation](https://www.volcengine.com/docs/82379/1263272) diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_claude.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_claude.md index b700f1644dd..fa831a05a9f 100644 --- a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_claude.md +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_claude.md @@ -1,10 +1,874 @@ --- Description: "" -date: "2025-03-19" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - claude weight: 0 --- -you can use LLM served by claude. +A Claude model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. + +## Features + +- Implements `github.com/cloudwego/eino/components/model.Model` +- Easy integration with Eino's model system +- Configurable model parameters +- Support for chat completion +- Support for streaming responses +- Custom response parsing support +- Flexible model configuration + +## Installation + +```bash +go get github.com/cloudwego/eino-ext/components/model/claude@latest +``` + +## Quick Start + +Here's a quick example of how to use the Claude model: + +```go +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/claude" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + if apiKey == "" { + log.Fatal("CLAUDE_API_KEY environment variable is not set") + } + + var baseURLPtr *string = nil + if len(baseURL) > 0 { + baseURLPtr = &baseURL + } + + // Create a Claude model + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: baseURLPtr, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + messages := []*schema.Message{ + { + Role: schema.System, + Content: "You are a helpful AI assistant. Be concise in your responses.", + }, + { + Role: schema.User, + Content: "What is the capital of France?", + }, + } + + resp, err := cm.Generate(ctx, messages, claude.WithThinking(&claude.Thinking{ + Enable: true, + BudgetTokens: 1024, + })) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + + thinking, ok := claude.GetThinking(resp) + fmt.Printf("Thinking(have: %v): %s\n", ok, thinking) + fmt.Printf("Assistant: %s\n", resp.Content) + if resp.ResponseMeta != nil && resp.ResponseMeta.Usage != nil { + fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total)\n", + resp.ResponseMeta.Usage.PromptTokens, + resp.ResponseMeta.Usage.CompletionTokens, + resp.ResponseMeta.Usage.TotalTokens) + } +} + +``` + +## Configuration + +The model can be configured using the `claude.ChatModelConfig` struct: + +```go +type Config struct { + // ByBedrock indicates whether to use Bedrock Service + // Required for Bedrock + ByBedrock bool + + // AccessKey is your Bedrock API Access key + // Obtain from: https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html + // Optional for Bedrock + AccessKey string + + // SecretAccessKey is your Bedrock API Secret Access key + // Obtain from: https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html + // Optional for Bedrock + SecretAccessKey string + + // SessionToken is your Bedrock API Session Token + // Obtain from: https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html + // Optional for Bedrock + SessionToken string + + // Profile is your Bedrock API AWS profile + // This parameter is ignored if AccessKey and SecretAccessKey are provided + // Obtain from: https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html + // Optional for Bedrock + Profile string + + // Region is your Bedrock API region + // Obtain from: https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html + // Optional for Bedrock + Region string + + // BaseURL is the custom API endpoint URL + // Use this to specify a different API endpoint, e.g., for proxies or enterprise setups + // Optional. Example: "https://custom-claude-api.example.com" + BaseURL *string + + // APIKey is your Anthropic API key + // Obtain from: https://console.anthropic.com/account/keys + // Required + APIKey string + + // Model specifies which Claude model to use + // Required + Model string + + // MaxTokens limits the maximum number of tokens in the response + // Range: 1 to model's context length + // Required. Example: 2000 for a medium-length response + MaxTokens int + + // Temperature controls randomness in responses + // Range: [0.0, 1.0], where 0.0 is more focused and 1.0 is more creative + // Optional. Example: float32(0.7) + Temperature *float32 + + // TopP controls diversity via nucleus sampling + // Range: [0.0, 1.0], where 1.0 disables nucleus sampling + // Optional. Example: float32(0.95) + TopP *float32 + + // TopK controls diversity by limiting the top K tokens to sample from + // Optional. Example: int32(40) + TopK *int32 + + // StopSequences specifies custom stop sequences + // The model will stop generating when it encounters any of these sequences + // Optional. Example: []string{"\n\nHuman:", "\n\nAssistant:"} + StopSequences []string + + Thinking *Thinking + + // HTTPClient specifies the client to send HTTP requests. + HTTPClient *http.Client `json:"http_client"` + + DisableParallelToolUse *bool `json:"disable_parallel_tool_use"` +} +``` + + + + + +## examples + +### generate + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/claude" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + if apiKey == "" { + log.Fatal("CLAUDE_API_KEY environment variable is not set") + } + + var baseURLPtr *string = nil + if len(baseURL) > 0 { + baseURLPtr = &baseURL + } + + // Create a Claude model + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: baseURLPtr, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + messages := []*schema.Message{ + { + Role: schema.System, + Content: "You are a helpful AI assistant. Be concise in your responses.", + }, + { + Role: schema.User, + Content: "What is the capital of France?", + }, + } + + resp, err := cm.Generate(ctx, messages, claude.WithThinking(&claude.Thinking{ + Enable: true, + BudgetTokens: 1024, + })) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + + thinking, ok := claude.GetThinking(resp) + fmt.Printf("Thinking(have: %v): %s\n", ok, thinking) + fmt.Printf("Assistant: %s\n", resp.Content) + if resp.ResponseMeta != nil && resp.ResponseMeta.Usage != nil { + fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total)\n", + resp.ResponseMeta.Usage.PromptTokens, + resp.ResponseMeta.Usage.CompletionTokens, + resp.ResponseMeta.Usage.TotalTokens) + } +} + +``` + +### generate_with_image + +```go + +package main + +import ( + "context" + "encoding/base64" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/claude" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + if apiKey == "" { + log.Fatal("CLAUDE_API_KEY environment variable is not set") + } + + var baseURLPtr *string = nil + if len(baseURL) > 0 { + baseURLPtr = &baseURL + } + + // Create a Claude model + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: baseURLPtr, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + imageBinary, err := os.ReadFile("examples/test.jpg") + if err != nil { + log.Fatalf("read file failed, err=%v", err) + } + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "What do you see in this image?", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: of(base64.StdEncoding.EncodeToString(imageBinary)), + MIMEType: "image/jpeg", + }, + }, + }, + }, + }, + }) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + fmt.Printf("Assistant: %s\n", resp.Content) +} + +func of[T any](v T) *T { + return &v +} + +``` + +### stream + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/claude" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + if apiKey == "" { + log.Fatal("CLAUDE_API_KEY environment variable is not set") + } + + var baseURLPtr *string = nil + if len(baseURL) > 0 { + baseURLPtr = &baseURL + } + + // Create a Claude model + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: baseURLPtr, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + messages := []*schema.Message{ + schema.SystemMessage("You are a helpful AI assistant. Be concise in your responses."), + { + Role: schema.User, + Content: "Write a short poem about spring, word by word.", + }, + } + + stream, err := cm.Stream(ctx, messages, claude.WithThinking(&claude.Thinking{ + Enable: true, + BudgetTokens: 1024, + })) + if err != nil { + log.Printf("Stream error: %v", err) + return + } + isFirstThinking := false + isFirstContent := false + + fmt.Print("Assistant: ----------\n") + for { + resp, err := stream.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Printf("Stream receive error: %v", err) + return + } + + thinkingContent, ok := claude.GetThinking(resp) + if ok { + if !isFirstThinking { + isFirstThinking = true + fmt.Print("\nThinking: ----------\n") + } + fmt.Print(thinkingContent) + } + + if len(resp.Content) > 0 { + if !isFirstContent { + isFirstContent = true + fmt.Print("\nContent: ----------\n") + } + fmt.Print(resp.Content) + } + } + fmt.Println("\n----------") +} + +``` + +### intent_tool + +```go + +package main + +import ( + "context" + "fmt" + + "io" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/claude" + "github.com/cloudwego/eino/schema" + "github.com/eino-contrib/jsonschema" + orderedmap "github.com/wk8/go-ordered-map/v2" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + if apiKey == "" { + log.Fatal("CLAUDE_API_KEY environment variable is not set") + } + + var baseURLPtr *string = nil + if len(baseURL) > 0 { + baseURLPtr = &baseURL + } + + // Create a Claude model + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: baseURLPtr, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + _, err = cm.WithTools([]*schema.ToolInfo{ + { + Name: "get_weather", + Desc: "Get current weather information for a city", + ParamsOneOf: schema.NewParamsOneOfByJSONSchema(&jsonschema.Schema{ + Type: "object", + Properties: orderedmap.New[string, *jsonschema.Schema](orderedmap.WithInitialData[string, *jsonschema.Schema]( + orderedmap.Pair[string, *jsonschema.Schema]{ + Key: "city", + Value: &jsonschema.Schema{ + Type: "string", + Description: "The city name", + }, + }, + orderedmap.Pair[string, *jsonschema.Schema]{ + Key: "unit", + Value: &jsonschema.Schema{ + Type: "string", + Enum: []interface{}{"celsius", "fahrenheit"}, + }, + }, + )), + Required: []string{"city"}, + }), + }, + }) + if err != nil { + log.Printf("Bind tools error: %v", err) + return + } + + streamResp, err := cm.Stream(ctx, []*schema.Message{ + schema.SystemMessage("You are a helpful AI assistant. Be concise in your responses."), + schema.UserMessage("call 'get_weather' to query what's the weather like in Paris today? Please use Celsius."), + }) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + + msgs := make([]*schema.Message, 0) + for { + msg, err := streamResp.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Fatalf("Stream receive error: %v", err) + } + msgs = append(msgs, msg) + } + resp, err := schema.ConcatMessages(msgs) + if err != nil { + log.Fatalf("Concat error: %v", err) + } + + fmt.Printf("assistant content:\n %v\n----------\n", resp.Content) + if len(resp.ToolCalls) > 0 { + fmt.Printf("Function called: %s\n", resp.ToolCalls[0].Function.Name) + fmt.Printf("Arguments: %s\n", resp.ToolCalls[0].Function.Arguments) + + weatherResp, err := cm.Generate(ctx, []*schema.Message{ + schema.UserMessage("What's the weather like in Paris today? Please use Celsius."), + resp, + schema.ToolMessage(`{"temperature": 18, "condition": "sunny"}`, resp.ToolCalls[0].ID), + }) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + fmt.Printf("Final response: %s\n", weatherResp.Content) + } +} + +``` + +### claude_prompt_cache + +```go + +package main + +import ( + "context" + "log" + "os" + "time" + + "github.com/cloudwego/eino-ext/components/model/claude" + "github.com/cloudwego/eino/components/model" + "github.com/cloudwego/eino/schema" +) + +func main() { + systemCache() + //toolInfoCache() + //sessionCache() +} + +func systemCache() { + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + + ctx := context.Background() + + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: &baseURL, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + breakpoint := claude.SetMessageBreakpoint(&schema.Message{ + Role: schema.System, + Content: "The film Dongji Rescue, based on a true historical event, tells the story of Chinese fishermen braving turbulent seas to save strangers — a testament to the nation's capacity to create miracles in the face of adversity.\n\nEighty-three years ago, in 1942, the Japanese military seized the Lisbon Maru ship to transport 1,816 British prisoners of war from Hong Kong to Japan. Passing through the waters near Zhoushan, Zhejiang province, it was torpedoed by a US submarine. Fishermen from Zhoushan rescued 384 prisoners of war and hid them from Japanese search parties.\n\nActor Zhu Yilong, in an exclusive interview with China Daily, said he was deeply moved by the humanity shown in such extreme conditions when he accepted his role. \"This historical event proves that in dire circumstances, Chinese people can extend their goodness to protect both themselves and others,\" he said.\n\nLast Friday, a themed event for Dongji Rescue was held in Zhoushan, a city close to where the Lisbon Maru sank. Descendants of the British POWs and the rescuing fishermen gathered to watch the film and share their reflections.\n\nIn the film, the British POWs are aided by a group of fishermen from Dongji Island, whose courage and compassion cut a path through treacherous waves. After the screening, many descendants were visibly moved.\n\n\"I want to express my deepest gratitude to the Chinese people and the descendants of Chinese fishermen. When the film is released in the UK, I will bring my family and friends to watch it. Heroic acts like this deserve to be known worldwide,\" said Denise Wynne, a descendant of a British POW.\n\n\"I felt the profound friendship between the Chinese and British people through the film,\" said Li Hui, a descendant of a rescuer. Many audience members were brought to tears — some by the fishermen's bravery, others by the enduring spirit of \"never abandoning those in peril\".\n\n\"In times of sea peril, rescue is a must,\" said Wu Buwei, another rescuer's descendant, noting that this value has long been a part of Dongji fishermen's traditions.\n\nCoincidentally, on the morning of Aug 5, a real-life rescue unfolded on the shores of Dongfushan in Dongji town. Two tourists from Shanghai fell into the sea and were swept away by powerful waves. Without hesitation, two descendants of Dongji fishermen — Wang Yubin and Yang Xiaoping — leaped in to save them, braving a wind force of 9 to 10 before pulling them to safety.\n\n\"Our forebearers once rescued many British POWs here. As their descendants, we should carry forward their bravery,\" Wang said. Both rescuers later joined the film's cast and crew at the Zhoushan event, sharing the stage with actors who portrayed the fishermen in a symbolic \"reunion across 83 years\".\n\nChinese actor Wu Lei, who plays A Dang in the film, expressed his respect for the two rescuing fishermen on-site. In the film, A Dang saves a British POW named Thomas Newman. Though they share no common language, they manage to connect. In an interview with China Daily, Wu said, \"They connected through a small globe. A Dang understood when Newman pointed out his home, the UK, on the globe.\"\n\nThe film's director Fei Zhenxiang noted that the Eastern Front in World War II is often overlooked internationally, despite China's 14 years of resistance tying down large numbers of Japanese troops. \"Every Chinese person is a hero, and their righteousness should not be forgotten,\" he said.\n\nGuan Hu, codirector, said, \"We hope that through this film, silent kindness can be seen by the world\", marking the 80th anniversary of victory in the Chinese People's War of Resistance Against Japanese Aggression (1931-45) and the World Anti-Fascist War.", + }) + + for i := 0; i < 2; i++ { + now := time.Now() + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "You are an AI assistant tasked with analyzing literary works. " + + "Your goal is to provide insightful commentary on themes.", + }, + breakpoint, + { + Role: schema.User, + Content: "Analyze the major themes in the content.", + }, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("time_consume=%f, output: \n%v", time.Now().Sub(now).Seconds(), resp) + } +} + +func toolInfoCache() { + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + + ctx := context.Background() + + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: &baseURL, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + breakpoint := claude.SetToolInfoBreakpoint(&schema.ToolInfo{ + Name: "get_time", + Desc: "Get the current time in a given time zone", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "timezone": { + Required: true, + Type: "string", + Desc: "The IANA time zone name, e.g. America/Los_Angeles", + }, + }, + )}, + ) + + mockTools := []*schema.ToolInfo{ + { + Name: "get_weather", + Desc: "Get the current weather in a given location", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "location": { + Type: "string", + Desc: "The city and state, e.g. San Francisco, CA", + }, + "unit": { + Type: "string", + Enum: []string{"celsius", "fahrenheit"}, + Desc: "The unit of temperature, either celsius or fahrenheit", + }, + }, + ), + }, + { + Name: "mock_tool_1", + Desc: "The film Dongji Rescue, based on a true historical event, tells the story of Chinese fishermen braving turbulent seas to save strangers — a testament to the nation's capacity to create miracles in the face of adversity.\n\nEighty-three years ago, in 1942, the Japanese military seized the Lisbon Maru ship to transport 1,816 British prisoners of war from Hong Kong to Japan. Passing through the waters near Zhoushan, Zhejiang province, it was torpedoed by a US submarine. Fishermen from Zhoushan rescued 384 prisoners of war and hid them from Japanese search parties.\n\nActor Zhu Yilong, in an exclusive interview with China Daily, said he was deeply moved by the humanity shown in such extreme conditions when he accepted his role. \"This historical event proves that in dire circumstances, Chinese people can extend their goodness to protect both themselves and others,\" he said.\n\nLast Friday, a themed event for Dongji Rescue was held in Zhoushan, a city close to where the Lisbon Maru sank. Descendants of the British POWs and the rescuing fishermen gathered to watch the film and share their reflections.\n\nIn the film, the British POWs are aided by a group of fishermen from Dongji Island, whose courage and compassion cut a path through treacherous waves. After the screening, many descendants were visibly moved.\n\n\"I want to express my deepest gratitude to the Chinese people and the descendants of Chinese fishermen. When the film is released in the UK, I will bring my family and friends to watch it. Heroic acts like this deserve to be known worldwide,\" said Denise Wynne, a descendant of a British POW.\n\n\"I felt the profound friendship between the Chinese and British people through the film,\" said Li Hui, a descendant of a rescuer. Many audience members were brought to tears — some by the fishermen's bravery, others by the enduring spirit of \"never abandoning those in peril\".\n\n\"In times of sea peril, rescue is a must,\" said Wu Buwei, another rescuer's descendant, noting that this value has long been a part of Dongji fishermen's traditions.\n\nCoincidentally, on the morning of Aug 5, a real-life rescue unfolded on the shores of Dongfushan in Dongji town. Two tourists from Shanghai fell into the sea and were swept away by powerful waves. Without hesitation, two descendants of Dongji fishermen — Wang Yubin and Yang Xiaoping — leaped in to save them, braving a wind force of 9 to 10 before pulling them to safety.\n\n\"Our forebearers once rescued many British POWs here. As their descendants, we should carry forward their bravery,\" Wang said. Both rescuers later joined the film's cast and crew at the Zhoushan event, sharing the stage with actors who portrayed the fishermen in a symbolic \"reunion across 83 years\".\n\nChinese actor Wu Lei, who plays A Dang in the film, expressed his respect for the two rescuing fishermen on-site. In the film, A Dang saves a British POW named Thomas Newman. Though they share no common language, they manage to connect. In an interview with China Daily, Wu said, \"They connected through a small globe. A Dang understood when Newman pointed out his home, the UK, on the globe.\"\n\nThe film's director Fei Zhenxiang noted that the Eastern Front in World War II is often overlooked internationally, despite China's 14 years of resistance tying down large numbers of Japanese troops. \"Every Chinese person is a hero, and their righteousness should not be forgotten,\" he said.\n\nGuan Hu, codirector, said, \"We hope that through this film, silent kindness can be seen by the world\", marking the 80th anniversary of victory in the Chinese People's War of Resistance Against Japanese Aggression (1931-45) and the World Anti-Fascist War.", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "mock_param": { + Required: true, + Type: "string", + Desc: "mock parameter", + }, + }, + ), + }, + { + Name: "mock_tool_2", + Desc: "For the administration, the biggest gain was Japan's commitment to investing $550 billion in the United States, particularly with funds to be directed to sectors deemed strategic, ranging from semiconductors and pharmaceuticals to energy infrastructure and critical minerals.The White House highlighted that the deal represents what it called the \"single largest foreign investment commitment ever secured by any country.\" It touted the agreement as a \"strategic realignment of the U.S.-Japan economic relationship\" that will advance the mutual interests of both countries.", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "mock_param": { + Required: true, + Type: "string", + Desc: "Bessent said it will check the extent of Japan's compliance every quarter and \"if the president is unhappy, then they will boomerang back to the 25 percent tariff rates, both on cars and the rest of their products.\"", + }, + }, + ), + }, + { + Name: "mock_tool_3", + Desc: "Of the amount, up to 100,000 tons is imported for direct consumption. Most of the remainder is used for animal feed and processing into other food products. Any rice imported to Japan beyond the special quota is subject to a tariff of 341 yen ($2.3) per kilogram.\n\n\n In fiscal 2024, the United States shipped roughly 346,000 tons of rice to Japan, compared with 286,000 tons exported by Thailand and 70,000 tons by Australia, under the quota system, according to Japan's farm ministry.", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "mock_param": { + Required: true, + Type: "string", + Desc: "Currently, Japan imports about 770,000 tons of tariff-free rice a year under the WTO framework, with the United States the No. 1 exporter, accounting for nearly half of the total, followed by Thailand and Australia.", + }, + }, + ), + }, + breakpoint, + } + + chatModelWithTools, err := cm.WithTools(mockTools) + if err != nil { + log.Fatalf("WithTools failed, err=%v", err) + } + + for i := 0; i < 2; i++ { + now := time.Now() + + resp, err := chatModelWithTools.Generate(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "You are a tool calling assistant.", + }, + { + Role: schema.User, + Content: "Mock the get_weather tool input yourself, and then call get_weather", + }, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + log.Printf("time_consume=%f, output: \n%v", time.Now().Sub(now).Seconds(), resp) + } +} + +func sessionCache() { + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + + ctx := context.Background() + + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: &baseURL, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + opts := []model.Option{ + claude.WithEnableAutoCache(true), + } + + mockTools := []*schema.ToolInfo{ + { + Name: "get_weather", + Desc: "Get the current weather in a given location", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "location": { + Type: "string", + Desc: "The city and state, e.g. San Francisco, CA", + }, + "unit": { + Type: "string", + Enum: []string{"celsius", "fahrenheit"}, + Desc: "The unit of temperature, either celsius or fahrenheit", + }, + }, + ), + }, + } + + chatModelWithTools, err := cm.WithTools(mockTools) + if err != nil { + log.Fatalf("WithTools failed, err=%v", err) + return + } + + input := []*schema.Message{ + schema.SystemMessage("You are a tool calling assistant."), + schema.UserMessage("What tools can you call?"), + } + + resp, err := chatModelWithTools.Generate(ctx, input, opts...) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + return + } + log.Printf("output_1: \n%v\n\n", resp) + + input = append(input, resp, schema.UserMessage("What am I asking last time?")) + + resp, err = chatModelWithTools.Generate(ctx, input, opts...) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + return + } + + log.Printf("output_2: \n%v\n\n", resp) +} + +``` + + + +## For More Details + +- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) +- [Claude Documentation](https://docs.claude.com/en/api/messages) diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_deepseek.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_deepseek.md index 00b83f484df..c58934a760e 100644 --- a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_deepseek.md +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_deepseek.md @@ -1,15 +1,13 @@ --- Description: "" -date: "2025-03-19" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - deepseek weight: 0 --- -## DeepSeek Model - -A DeepSeek model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `Model` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +A DeepSeek model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. ## Features @@ -39,11 +37,10 @@ import ( "fmt" "log" "os" - - "github.com/cloudwego/eino/components/model" - "github.com/cloudwego/eino/schema" + "time" "github.com/cloudwego/eino-ext/components/model/deepseek" + "github.com/cloudwego/eino/schema" ) func main() { @@ -52,12 +49,12 @@ func main() { if apiKey == "" { log.Fatal("DEEPSEEK_API_KEY environment variable is not set") } - - // 创建 deepseek 模型 cm, err := deepseek.NewChatModel(ctx, &deepseek.ChatModelConfig{ - APIKey: apiKey, - Model: "deepseek-reasoner", - MaxTokens: 2000, + APIKey: apiKey, + Model: os.Getenv("MODEL_NAME"), + BaseURL: "https://api.deepseek.com/beta", + + }) if err != nil { log.Fatal(err) @@ -84,7 +81,7 @@ func main() { if !ok { fmt.Printf("Unexpected: non-reasoning") } else { - fmt.Printf("Resoning Content: %s\n", reasoning) + fmt.Printf("Reasoning Content: %s\n", reasoning) } fmt.Printf("Assistant: %s\n", resp.Content) if resp.ResponseMeta != nil && resp.ResponseMeta.Usage != nil { @@ -93,6 +90,7 @@ func main() { resp.ResponseMeta.Usage.CompletionTokens, resp.ResponseMeta.Usage.TotalTokens) } + } ``` @@ -101,64 +99,389 @@ func main() { The model can be configured using the `deepseek.ChatModelConfig` struct: ```go + type ChatModelConfig struct { -// APIKey is your authentication key -// Required -APIKey string `json:"api_key"` - -// Timeout specifies the maximum duration to wait for API responses -// Optional. Default: 5 minutes -Timeout time.Duration `json:"timeout"` - -// BaseURL is your custom deepseek endpoint url -// Optional. Default: https://api.deepseek.com/ -BaseURL string `json:"base_url"` - -// The following fields correspond to DeepSeek's chat API parameters -// Ref: https://api-docs.deepseek.com/api/create-chat-completion - -// Model specifies the ID of the model to use -// Required -Model string `json:"model"` - -// MaxTokens limits the maximum number of tokens that can be generated in the chat completion -// Range: [1, 8192]. -// Optional. Default: 4096 -MaxTokens int `json:"max_tokens,omitempty"` - -// Temperature specifies what sampling temperature to use -// Generally recommend altering this or TopP but not both. -// Range: [0.0, 2.0]. Higher values make output more random -// Optional. Default: 1.0 -Temperature float32 `json:"temperature,omitempty"` - -// TopP controls diversity via nucleus sampling -// Generally recommend altering this or Temperature but not both. -// Range: [0.0, 1.0]. Lower values make output more focused -// Optional. Default: 1.0 -TopP float32 `json:"top_p,omitempty"` - -// Stop sequences where the API will stop generating further tokens -// Optional. Example: []string{"\n", "User:"} -Stop []string `json:"stop,omitempty"` - -// PresencePenalty prevents repetition by penalizing tokens based on presence -// Range: [-2.0, 2.0]. Positive values increase likelihood of new topics -// Optional. Default: 0 -PresencePenalty float32 `json:"presence_penalty,omitempty"` - -// ResponseFormat specifies the format of the model's response -// Optional. Use for structured outputs -ResponseFormatType ResponseFormatType `json:"response_format_type,omitempty"` - -// FrequencyPenalty prevents repetition by penalizing tokens based on frequency -// Range: [-2.0, 2.0]. Positive values decrease likelihood of repetition -// Optional. Default: 0 -FrequencyPenalty float32 `json:"frequency_penalty,omitempty"` + // APIKey is your authentication key + // Required + APIKey string `json:"api_key"` + + // Timeout specifies the maximum duration to wait for API responses + // Optional. Default: 5 minutes + Timeout time.Duration `json:"timeout"` + + // HTTPClient specifies the client to send HTTP requests. + // Optional. Default http.DefaultClient + HTTPClient *http.Client `json:"http_client"` + + // BaseURL is your custom deepseek endpoint url + // Optional. Default: https://api.deepseek.com/ + BaseURL string `json:"base_url"` + + // Path sets the path for the API request. Defaults to "chat/completions", if not set. + // Example usages would be "/c/chat/" or any http after the baseURL extension + // Path 用于设置 API 请求的路径。如果未设置，则默认为 "chat/completions"。 + // 用法示例可以是 "/c/chat/" 或 baseURL 之后的任何 http 路径。 + Path string `json:"path"` + + // The following fields correspond to DeepSeek's chat API parameters + // Ref: https://api-docs.deepseek.com/api/create-chat-completion + + // Model specifies the ID of the model to use + // Required + Model string `json:"model"` + + // MaxTokens limits the maximum number of tokens that can be generated in the chat completion + // Range: [1, 8192]. + // Optional. Default: 4096 + MaxTokens int `json:"max_tokens,omitempty"` + + // Temperature specifies what sampling temperature to use + // Generally recommend altering this or TopP but not both. + // Range: [0.0, 2.0]. Higher values make output more random + // Optional. Default: 1.0 + Temperature float32 `json:"temperature,omitempty"` + + // TopP controls diversity via nucleus sampling + // Generally recommend altering this or Temperature but not both. + // Range: [0.0, 1.0]. Lower values make output more focused + // Optional. Default: 1.0 + TopP float32 `json:"top_p,omitempty"` + + // Stop sequences where the API will stop generating further tokens + // Optional. Example: []string{"\n", "User:"} + Stop []string `json:"stop,omitempty"` + + // PresencePenalty prevents repetition by penalizing tokens based on presence + // Range: [-2.0, 2.0]. Positive values increase likelihood of new topics + // Optional. Default: 0 + PresencePenalty float32 `json:"presence_penalty,omitempty"` + + // ResponseFormat specifies the format of the model's response + // Optional. Use for structured outputs + ResponseFormatType ResponseFormatType `json:"response_format_type,omitempty"` + + // FrequencyPenalty prevents repetition by penalizing tokens based on frequency + // Range: [-2.0, 2.0]. Positive values decrease likelihood of repetition + // Optional. Default: 0 + FrequencyPenalty float32 `json:"frequency_penalty,omitempty"` + + // LogProbs specifies whether to return log probabilities of the output tokens. + LogProbs bool `json:"log_probs"` + + // TopLogProbs specifies the number of most likely tokens to return at each token position, each with an associated log probability. + TopLogProbs int `json:"top_log_probs"` +} + +``` + + + + + +## examples + +### generate + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + "time" + + "github.com/cloudwego/eino-ext/components/model/deepseek" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("DEEPSEEK_API_KEY") + if apiKey == "" { + log.Fatal("DEEPSEEK_API_KEY environment variable is not set") + } + + cm, err := deepseek.NewChatModel(ctx, &deepseek.ChatModelConfig{ + APIKey: apiKey, + Model: os.Getenv("MODEL_NAME"), + BaseURL: "https://api.deepseek.com/beta", + Timeout: 30 * time.Second, + }) + if err != nil { + log.Fatal(err) + } + + messages := []*schema.Message{ + { + Role: schema.System, + Content: "You are a helpful AI assistant. Be concise in your responses.", + }, + { + Role: schema.User, + Content: "What is the capital of France?", + }, + } + + resp, err := cm.Generate(ctx, messages) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + + reasoning, ok := deepseek.GetReasoningContent(resp) + if !ok { + fmt.Printf("Unexpected: non-reasoning") + } else { + fmt.Printf("Reasoning Content: %s\n", reasoning) + } + fmt.Printf("Assistant: %s\n", resp.Content) + if resp.ResponseMeta != nil && resp.ResponseMeta.Usage != nil { + fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total)\n", + resp.ResponseMeta.Usage.PromptTokens, + resp.ResponseMeta.Usage.CompletionTokens, + resp.ResponseMeta.Usage.TotalTokens) + } + } + ``` +### generate_with_prefix + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + "time" + + "github.com/cloudwego/eino-ext/components/model/deepseek" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("DEEPSEEK_API_KEY") + if apiKey == "" { + log.Fatal("DEEPSEEK_API_KEY environment variable is not set") + } + cm, err := deepseek.NewChatModel(ctx, &deepseek.ChatModelConfig{ + APIKey: apiKey, + Model: os.Getenv("MODEL_NAME"), + BaseURL: "https://api.deepseek.com/beta", + Timeout: 30 * time.Second, + }) + if err != nil { + log.Fatal(err) + } + + messages := []*schema.Message{ + schema.UserMessage("Please write quick sort code"), + schema.AssistantMessage("```python\n", nil), + } + deepseek.SetPrefix(messages[1]) + + result, err := cm.Generate(ctx, messages) + if err != nil { + log.Printf("Generate error: %v", err) + } + + reasoningContent, ok := deepseek.GetReasoningContent(result) + if !ok { + fmt.Printf("No reasoning content") + } else { + fmt.Printf("Reasoning: %v\n", reasoningContent) + } + fmt.Printf("Content: %v\n", result) + +} + +``` + +### stream + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + "time" + + "github.com/cloudwego/eino-ext/components/model/deepseek" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("DEEPSEEK_API_KEY") + if apiKey == "" { + log.Fatal("DEEPSEEK_API_KEY environment variable is not set") + } + cm, err := deepseek.NewChatModel(ctx, &deepseek.ChatModelConfig{ + APIKey: apiKey, + Model: os.Getenv("MODEL_NAME"), + BaseURL: "https://api.deepseek.com/beta", + Timeout: 30 * time.Second, + }) + if err != nil { + log.Fatal(err) + } + + messages := []*schema.Message{ + { + Role: schema.User, + Content: "Write a short poem about spring, word by word.", + }, + } + + stream, err := cm.Stream(ctx, messages) + if err != nil { + log.Printf("Stream error: %v", err) + return + } + + fmt.Print("Assistant: ") + for { + resp, err := stream.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Printf("Stream receive error: %v", err) + return + } + if reasoning, ok := deepseek.GetReasoningContent(resp); ok { + fmt.Printf("Reasoning Content: %s\n", reasoning) + } + if len(resp.Content) > 0 { + fmt.Printf("Content: %s\n", resp.Content) + } + if resp.ResponseMeta != nil && resp.ResponseMeta.Usage != nil { + fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total)\n", + resp.ResponseMeta.Usage.PromptTokens, + resp.ResponseMeta.Usage.CompletionTokens, + resp.ResponseMeta.Usage.TotalTokens) + } + } +} + +``` + +### intent_tool + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + "time" + + "github.com/cloudwego/eino-ext/components/model/deepseek" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("DEEPSEEK_API_KEY") + if apiKey == "" { + log.Fatal("DEEPSEEK_API_KEY environment variable is not set") + } + cm, err := deepseek.NewChatModel(ctx, &deepseek.ChatModelConfig{ + APIKey: apiKey, + Model: os.Getenv("MODEL_NAME"), + BaseURL: "https://api.deepseek.com/beta", + Timeout: 30 * time.Second, + }) + if err != nil { + log.Fatal(err) + } + + _, err = cm.WithTools([]*schema.ToolInfo{ + { + Name: "user_company", + Desc: "Retrieve the user's company and position based on their name and email.", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": {Type: "string", Desc: "user's name"}, + "email": {Type: "string", Desc: "user's email"}}), + }, { + Name: "user_salary", + Desc: "Retrieve the user's salary based on their name and email.\n", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": {Type: "string", Desc: "user's name"}, + "email": {Type: "string", Desc: "user's email"}, + }), + }}) + if err != nil { + log.Fatalf("BindTools of deepseek failed, err=%v", err) + } + resp, err := cm.Generate(ctx, []*schema.Message{{ + Role: schema.System, + Content: "As a real estate agent, provide relevant property information based on the user's salary and job using the user_company and user_salary APIs. An email address is required.", + }, { + Role: schema.User, + Content: "My name is John and my email is john@abc.com，Please recommend some houses that suit me.", + }}) + if err != nil { + log.Fatalf("Generate of deepseek failed, err=%v", err) + } + fmt.Printf("output: \n%v", resp) + + streamResp, err := cm.Stream(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "As a real estate agent, provide relevant property information based on the user's salary and job using the user_company and user_salary APIs. An email address is required.", + }, { + Role: schema.User, + Content: "My name is John and my email is john@abc.com，Please recommend some houses that suit me.", + }, + }) + if err != nil { + log.Fatalf("Stream of deepseek failed, err=%v", err) + } + var messages []*schema.Message + for { + chunk, err := streamResp.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Fatalf("Recv of streamResp failed, err=%v", err) + } + messages = append(messages, chunk) + } + resp, err = schema.ConcatMessages(messages) + if err != nil { + log.Fatalf("ConcatMessages of deepseek failed, err=%v", err) + } + fmt.Printf("stream output: \n%v", resp) + +} + +``` + + + + ## For More Details -- [Eino Documentation](https://github.com/cloudwego/eino) +- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) - [DeepSeek Documentation](https://api-docs.deepseek.com/api/create-chat-completion) diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_gemini.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_gemini.md index 351d5e9c436..2cd01849d88 100644 --- a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_gemini.md +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_gemini.md @@ -1,10 +1,845 @@ --- Description: "" -date: "2025-03-19" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - gemini weight: 0 --- -you can use LLM served by gemini. +A Google Gemini implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. + +## Features + +- Implements `github.com/cloudwego/eino/components/model.Model` +- Easy integration with Eino's model system +- Configurable model parameters +- Support for chat completion +- Support for streaming responses +- Custom response parsing support +- Flexible model configuration +- Caching support for generated responses + +## Installation + +```bash +go get github.com/cloudwego/eino-ext/components/model/gemini@latest +``` + +## Quick start + +Here's a quick example of how to use the Gemini model: + +```go +package main + +import ( + "context" + "fmt" + "log" + "os" + + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" + "github.com/cloudwego/eino/schema" +) + +func main() { + apiKey := os.Getenv("GEMINI_API_KEY") + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: apiKey, + }) + if err != nil { + log.Fatalf("NewClient of gemini failed, err=%v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Client: client, + Model: "gemini-2.5-flash", + ThinkingConfig: &genai.ThinkingConfig{ + IncludeThoughts: true, + ThinkingBudget: nil, + }, + }) + if err != nil { + log.Fatalf("NewChatModel of gemini failed, err=%v", err) + } + + // If you are using a model that supports image understanding (e.g., gemini-2.5-flash-image-preview), + // you can provide both image and text input like this: + /* + image, err := os.ReadFile("./path/to/your/image.jpg") + if err != nil { + log.Fatalf("os.ReadFile failed, err=%v\n", err) + } + + imageStr := base64.StdEncoding.EncodeToString(image) + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "What do you see in this image?", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: &imageStr, + MIMEType: "image/jpeg", + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + }, + }, + }) + */ + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "What is the capital of France?", + }, + }) + if err != nil { + log.Fatalf("Generate error: %v", err) + } + + fmt.Printf("Assistant: %s\n", resp.Content) + if len(resp.ReasoningContent) > 0 { + fmt.Printf("ReasoningContent: %s\n", resp.ReasoningContent) + } +} +``` + +## Configuration + +The model can be configured using the `gemini.Config` struct: + +```go +type Config struct { + // Client is the Gemini API client instance + // Required for making API calls to Gemini + Client *genai.Client + + // Model specifies which Gemini model to use + // Examples: "gemini-pro", "gemini-pro-vision", "gemini-1.5-flash" + Model string + + // MaxTokens limits the maximum number of tokens in the response + // Optional. Example: maxTokens := 100 + MaxTokens *int + + // Temperature controls randomness in responses + // Range: [0.0, 1.0], where 0.0 is more focused and 1.0 is more creative + // Optional. Example: temperature := float32(0.7) + Temperature *float32 + + // TopP controls diversity via nucleus sampling + // Range: [0.0, 1.0], where 1.0 disables nucleus sampling + // Optional. Example: topP := float32(0.95) + TopP *float32 + + // TopK controls diversity by limiting the top K tokens to sample from + // Optional. Example: topK := int32(40) + TopK *int32 + + // ResponseSchema defines the structure for JSON responses + // Optional. Used when you want structured output in JSON format + ResponseSchema *openapi3.Schema + + // EnableCodeExecution allows the model to execute code + // Warning: Be cautious with code execution in production + // Optional. Default: false + EnableCodeExecution bool + + // SafetySettings configures content filtering for different harm categories + // Controls the model's filtering behavior for potentially harmful content + // Optional. + SafetySettings []*genai.SafetySetting + + ThinkingConfig *genai.ThinkingConfig + + // ResponseModalities specifies the modalities the model can return. + // Optional. + ResponseModalities [] + + MediaResolution genai.MediaResolution + + // Cache controls prefix cache settings for the model. + // Optional. used to CreatePrefixCache for reused inputs. + Cache *CacheConfig +} + +// CacheConfig controls prefix cache settings for the model. +type CacheConfig struct { + // TTL specifies how long cached resources remain valid (now + TTL). + TTL time.Duration `json:"ttl,omitempty"` + // ExpireTime sets the absolute expiration timestamp for cached resources. + ExpireTime time.Time `json:"expireTime,omitempty"` +} +``` + +## Caching + +This component supports two caching strategies to improve latency and reduce API calls: + +- Explicit caching (prefix cache): Build a reusable context from the system instruction, tools, and messages. Use `CreatePrefixCache` to create the cache and pass its name with `gemini.WithCachedContentName(...)` in subsequent requests. Configure TTL and absolute expiry via `CacheConfig` (`TTL`, `ExpireTime`). When a cached content is used, the request omits system instruction and tools and relies on the cached prefix. +- Implicit caching: Managed by Gemini itself. The service may reuse prior requests or responses automatically. Expiry and reuse are controlled by Gemini and cannot be configured. + +``` +toolInfoList := []*schema.ToolInfo{ + { + Name: "tool_a", + Desc: "desc", + ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{}), + }, +} +cacheInfo, _ := cm.CreatePrefixCache(ctx, []*schema.Message{ + { + Role: schema.System, + Content: `aaa`, + }, + { + Role: schema.User, + Content: `bbb`, + }, + }, model.WithTools(toolInfoList)) + + +msg, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "give a very short summary about this transcript", + }, + }, gemini.WithCachedContentName(cacheInfo.Name)) +``` + +The example above shows how to create a prefix cache and reuse it in a follow-up call. + +## examples + +### generate + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" + "github.com/cloudwego/eino/schema" +) + +func main() { + apiKey := os.Getenv("GEMINI_API_KEY") + modelName := os.Getenv("GEMINI_MODEL") + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: apiKey, + }) + if err != nil { + log.Fatalf("NewClient of gemini failed, err=%v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Client: client, + Model: modelName, + ThinkingConfig: &genai.ThinkingConfig{ + IncludeThoughts: true, + ThinkingBudget: nil, + }, + }) + if err != nil { + log.Fatalf("NewChatModel of gemini failed, err=%v", err) + } + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "What is the capital of France?", + }, + }) + if err != nil { + log.Fatalf("Generate error: %v", err) + } + + fmt.Printf("Assistant: %s\n", resp.Content) + if len(resp.ReasoningContent) > 0 { + fmt.Printf("ReasoningContent: %s\n", resp.ReasoningContent) + } +} + +``` + +### generate_with_image + +```go + +package main + +import ( + "context" + "encoding/base64" + "fmt" + "log" + "os" + + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" + "github.com/cloudwego/eino/schema" +) + +func main() { + apiKey := os.Getenv("GEMINI_API_KEY") + modelName := os.Getenv("GEMINI_MODEL") + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: apiKey, + }) + if err != nil { + log.Fatalf("NewClient of gemini failed, err=%v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Client: client, + Model: modelName, + }) + if err != nil { + log.Fatalf("NewChatModel of gemini failed, err=%v", err) + } + + image, err := os.ReadFile("./examples/generate_with_image/test.jpg") + if err != nil { + log.Fatalf("os.ReadFile failed, err=%v\n", err) + } + + imageStr := base64.StdEncoding.EncodeToString(image) + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "What do you see in this image?", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: &imageStr, + MIMEType: "image/jpeg", + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + }, + }, + }) + if err != nil { + log.Fatalf("Generate error: %v", err) + } + fmt.Printf("Assistant: %s\n", resp.Content) +} + +``` + +### generate_with_prefix_cache + +```go + +package main + +import ( + "context" + "encoding/base64" + "fmt" + "log" + "os" + + "github.com/bytedance/sonic" + "github.com/cloudwego/eino/components/model" + "github.com/cloudwego/eino/components/tool/utils" + "github.com/cloudwego/eino/schema" + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" +) + +func main() { + ctx := context.Background() + + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + }) + if err != nil { + log.Fatalf("genai.NewClient failed: %v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Model: os.Getenv("GEMINI_MODEL"), + Client: client, + }) + if err != nil { + log.Fatalf("gemini.NewChatModel failed: %v", err) + } + + type toolCallInput struct { + Answer int `json:"answer" jsonschema_description:"the answer of the question"` + } + answerTool, err := utils.InferTool("answer_to_user", + "answer to user", + func(ctx context.Context, in *toolCallInput) (string, error) { + return fmt.Sprintf("answer: %v", in.Answer), nil + }) + if err != nil { + log.Fatalf("utils.InferTool failed: %v", err) + } + + info, err := answerTool.Info(ctx) + if err != nil { + log.Fatalf("get tool info failed: %v", err) + } + + // this file is from gemini cache usage example + fileData, err := os.ReadFile("./a11.test.txt") + if err != nil { + log.Fatalf("os.ReadFile failed: %v", err) + } + + txtFileBase64 := base64.StdEncoding.EncodeToString(fileData) + cacheInfo, err := cm.CreatePrefixCache(ctx, []*schema.Message{ + { + Role: schema.System, + Content: `You are an expert at analyzing transcripts. +answer the question with the tool "answer_to_user" +always include the start_time and end_time of the transcript in the output`, + }, + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeFileURL, + File: &schema.MessageInputFile{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: &txtFileBase64, + MIMEType: "text/plain", + }, + }, + }, + }, + }, + }, model.WithTools([]*schema.ToolInfo{info}), model.WithToolChoice(schema.ToolChoiceForced)) + if err != nil { + log.Fatalf("CreatePrefixCache failed: %v", err) + } + + data, _ := sonic.MarshalIndent(cacheInfo, "", " ") + log.Printf("prefix cache info:\n%v\n", string(data)) + + msg, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "give a very short summary about this transcript", + }, + }, gemini.WithCachedContentName(cacheInfo.Name), + model.WithTools([]*schema.ToolInfo{info}), + model.WithToolChoice(schema.ToolChoiceForced)) + if err != nil { + log.Fatalf("Generate failed: %v", err) + } + msgData, _ := sonic.MarshalIndent(msg, "", " ") + log.Printf("model output:\n%v\n", string(msgData)) +} + +``` + +### stream + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" + "github.com/cloudwego/eino/schema" +) + +func main() { + apiKey := os.Getenv("GEMINI_API_KEY") + modelName := os.Getenv("GEMINI_MODEL") + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: apiKey, + }) + if err != nil { + log.Fatalf("NewClient of gemini failed, err=%v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Client: client, + Model: modelName, + ThinkingConfig: &genai.ThinkingConfig{ + IncludeThoughts: true, + ThinkingBudget: nil, + }, + }) + if err != nil { + log.Fatalf("NewChatModel of gemini failed, err=%v", err) + } + stream, err := cm.Stream(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "Write a short poem about spring.", + }, + }) + if err != nil { + log.Fatalf("Stream error: %v", err) + } + + fmt.Println("Assistant: ") + for { + resp, err := stream.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Fatalf("Stream receive error: %v", err) + } + + fmt.Println("frame: ") + if len(resp.Content) > 0 { + fmt.Println("content: ", resp.Content) + } + if len(resp.ReasoningContent) > 0 { + fmt.Printf("ReasoningContent: %s\n", resp.ReasoningContent) + } + } + fmt.Println() +} + +``` + +### intent_tool + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" + "github.com/cloudwego/eino/schema" +) + +func main() { + apiKey := os.Getenv("GEMINI_API_KEY") + modelName := os.Getenv("GEMINI_MODEL") + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: apiKey, + }) + if err != nil { + log.Fatalf("NewClient of gemini failed, err=%v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Client: client, + Model: modelName, + ThinkingConfig: &genai.ThinkingConfig{ + IncludeThoughts: true, + ThinkingBudget: nil, + }, + }) + if err != nil { + log.Fatalf("NewChatModel of gemini failed, err=%v", err) + } + err = cm.BindTools([]*schema.ToolInfo{ + { + Name: "book_recommender", + Desc: "Recommends books based on user preferences and provides purchase links", + ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{ + "genre": { + Type: "string", + Desc: "Preferred book genre", + Enum: []string{"fiction", "sci-fi", "mystery", "biography", "business"}, + }, + "max_pages": { + Type: "integer", + Desc: "Maximum page length (0 for no limit)", + }, + "min_rating": { + Type: "number", + Desc: "Minimum user rating (0-5 scale)", + }, + }), + }, + }) + if err != nil { + log.Fatalf("Bind tools error: %v", err) + } + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "Recommend business books with minimum 4.3 rating and max 350 pages", + }, + }) + if err != nil { + log.Fatalf("Generate error: %v", err) + } + + if len(resp.ToolCalls) > 0 { + fmt.Printf("Function called: \n") + if len(resp.ReasoningContent) > 0 { + fmt.Printf("ReasoningContent: %s\n", resp.ReasoningContent) + } + fmt.Println("Name: ", resp.ToolCalls[0].Function.Name) + fmt.Printf("Arguments: %s\n", resp.ToolCalls[0].Function.Arguments) + } else { + log.Printf("Function called without tool calls: %s\n", resp.Content) + } + + resp, err = cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "Recommend business books with minimum 4.3 rating and max 350 pages", + }, + resp, + { + Role: schema.Tool, + ToolCallID: resp.ToolCalls[0].ID, + Content: "{\"book name\":\"Microeconomics for Managers\"}", + }, + }) + if err != nil { + log.Fatalf("Generate error: %v", err) + } + fmt.Printf("Function call final result: %s\n", resp.Content) +} + +``` + +### image_generate + +```go + +package main + +import ( + "context" + "encoding/json" + "log" + "os" + + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" + "github.com/cloudwego/eino/schema" +) + +func main() { + apiKey := os.Getenv("GEMINI_API_KEY") + modelName := os.Getenv("GEMINI_MODEL") + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: apiKey, + }) + if err != nil { + log.Fatalf("NewClient of gemini failed, err=%v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Client: client, + Model: modelName, + ResponseModalities: []gemini.GeminiResponseModality{ + gemini.GeminiResponseModalityText, + gemini.GeminiResponseModalityImage, + }, + }) + if err != nil { + log.Fatalf("NewChatModel of gemini failed, err=%v", err) + } + + /* + The generated multimodal content is stored in the `AssistantGenMultiContent` field. + For this example, the resulting message will have a structure similar to this: + + resp := &schema.Message{ + Role: schema.Assistant, + AssistantGenMultiContent: []schema.MessageOutputPart{ + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageOutputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: &base64String, // The base64 encoded image data + MIMEType: "image/png", + }, + }, + }, + }, + } + */ + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "Generate an image of a cat", + }, + }, + }, + }) + if err != nil { + log.Fatalf("Generate error: %v", err) + } + log.Printf("\ngenerate output: \n") + respBody, _ := json.MarshalIndent(resp, " ", " ") + log.Printf(" body: %s\n", string(respBody)) +} + +``` + +### react + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/bytedance/sonic" + "github.com/cloudwego/eino/adk" + "github.com/cloudwego/eino/components/tool" + "github.com/cloudwego/eino/components/tool/utils" + "github.com/cloudwego/eino/compose" + "github.com/cloudwego/eino/schema" + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" +) + +func main() { + ctx := context.Background() + + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + }) + if err != nil { + log.Fatalf("genai.NewClient failed, err=%v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Model: os.Getenv("GEMINI_MODEL"), + Client: client, + }) + if err != nil { + log.Fatalf("gemini.NewChatModel failed, err=%v", err) + } + + type toolCallInput struct { + LastCount int `json:"last_count" jsonschema_description:"the last count"` + } + countsTool, err := utils.InferTool("count_tool_call", + "count the number of tool calls", + func(ctx context.Context, in *toolCallInput) (string, error) { + counts := in.LastCount + 1 + return fmt.Sprintf("tool call counts: %v", counts), nil + }) + if err != nil { + log.Fatalf("utils.InferTool failed, err=%v", err) + } + + agent, err := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{ + Name: "react_agent", + Description: "react_agent", + Instruction: `call count_tool_call 5 times, then say 'done'`, + Model: cm, + ToolsConfig: adk.ToolsConfig{ + ToolsNodeConfig: compose.ToolsNodeConfig{ + Tools: []tool.BaseTool{ + countsTool, + }, + }, + }, + }) + if err != nil { + log.Fatalf("adk.NewChatModelAgent failed, err=%v", err) + } + + iter := agent.Run(ctx, &adk.AgentInput{ + Messages: []adk.Message{ + { + Role: schema.User, + Content: "start to count", + }, + }, + }) + idx := 0 + for { + event, ok := iter.Next() + if !ok { + break + } + + if event.Err != nil { + log.Fatalf("agent.Run failed, err=%v", event.Err) + } + + msg, err_ := event.Output.MessageOutput.GetMessage() + if err_ != nil { + log.Fatalf("GetMessage failed, err=%v", err_) + } + + idx++ + msgData, _ := sonic.MarshalIndent(msg, "", " ") + log.Printf("\nmessage %v:\n%v\n", idx, string(msgData)) + } +} + +``` + + + +## For More Details + +- [Eino Documentation](https://github.com/cloudwego/eino) +- [Gemini API Documentation](https://ai.google.dev/api/generate-content?hl=zh-cn#v1beta.GenerateContentResponse) diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ollama.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ollama.md index bfeb8ae340d..fcc8b5da0bf 100644 --- a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ollama.md +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_ollama.md @@ -1,214 +1,449 @@ --- Description: "" -date: "2025-03-19" +date: "2025-12-02" lastmod: "" tags: [] -title: ChatModel - Ollama +title: ChatModel - ollama weight: 0 --- -## **Basic Introduction** +A Ollama model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. -The Ollama model is an implementation of the ChatModel interface, designed for interacting with the Ollama local LLM service. Ollama is an open-source local LLM runtime framework that supports various open-source models (such as Llama, Mistral, etc.), providing simple API interfaces and comprehensive performance monitoring. This component implements the [Eino: ChatModel Usage Instructions]([Eino: ChatModel guide](/docs/eino/core_modules/components/chat_model_guide)) +## Features -## **Usage** +- Implements `github.com/cloudwego/eino/components/model.Model` +- Easy integration with Eino's model system +- Configurable model parameters +- Support for chat completion +- Support for streaming responses +- Custom response parsing support +- Flexible model configuration -### **Component Initialization** +## Installation -The Ollama model is initialized through the `NewChatModel` function. The main configuration parameters are as follows: +```bash +go get github.com/cloudwego/eino-ext/components/model/ollama@latest +``` + +## Quick Start + +Here's a quick example of how to use the Ollama model: ```go +package main + import ( - "github.com/cloudwego/eino-ext/components/model/ollama" - "github.com/ollama/ollama/api" + "context" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ollama" ) -model, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ - // Basic Configuration - BaseURL: "http://localhost:11434", // Ollama service address - Timeout: 30 * time.Second, // Request timeout - - // Model Configuration - Model: "llama2", // Model name - Format: json.RawMessage(`"json"`), // Output format (optional) - KeepAlive: &keepAlive, // Keep-alive time - - // Model Parameters - Options: &api.Options{ - Runner: api.Runner{ - NumCtx: 4096, // Context window size - NumGPU: 1, // Number of GPUs - NumThread: 4, // Number of CPU threads - }, - Temperature: 0.7, // Temperature - TopP: 0.9, // Top-P sampling - TopK: 40, // Top-K sampling - Seed: 42, // Random seed - NumPredict: 100, // Maximum generation length - Stop: []string{}, // Stop words - RepeatPenalty: 1.1, // Repeat penalty - }, -}) +func main() { + ctx := context.Background() + modelName := os.Getenv("MODEL_NAME") + + chatModel, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ + BaseURL: "http://localhost:11434", + Model: modelName, + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v\n", err) + return + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "as a machine, how do you answer user's question?", + }, + }) + if err != nil { + log.Printf("Generate failed, err=%v\n", err) + return + } + + log.Printf("output: \n%v\n", resp) +} + ``` -### **Generating Conversations** +## Configuration + +The model can be configured using the `ollama.ChatModelConfig` struct: -Conversation generation supports both normal mode and streaming mode: ```go -// Normal mode -response, err := model.Generate(ctx, messages) +type ChatModelConfig struct { + BaseURL string `json:"base_url"` + Timeout time.Duration `json:"timeout"` // request timeout for http client + + // HTTPClient specifies the client to send HTTP requests. + // If HTTPClient is set, Timeout will not be used. + // Optional. Default &http.Client{Timeout: Timeout} + HTTPClient *http.Client `json:"http_client"` + + Model string `json:"model"` + Format json.RawMessage `json:"format"` + KeepAlive *time.Duration `json:"keep_alive"` -// Streaming mode -stream, err := model.Stream(ctx, messages) + Options *Options `json:"options"` + + Thinking *ThinkValue `json:"thinking"` +} + + +type Options struct { + Runner + + // NumKeep specifies the number of tokens from the prompt to retain when the context size is exceeded and tokens need to be trimmed. + NumKeep int `json:"num_keep,omitempty"` + // Seed sets the random number seed for the model. Using the same seed with the same parameters will produce the same output. + Seed int `json:"seed,omitempty"` + // NumPredict sets the maximum number of tokens to generate. + NumPredict int `json:"num_predict,omitempty"` + // TopK controls the diversity of the generated text by limiting the selection of tokens to the top k most likely tokens. + TopK int `json:"top_k,omitempty"` + // TopP, also known as nucleus sampling, is another way to control the diversity of the generated text. It filters out the least likely tokens whose cumulative probability is below a certain threshold. + TopP float32 `json:"top_p,omitempty"` + // MinP is a parameter that works with TopP to ensure that the generated text is not too constrained. It sets a minimum probability for a token to be considered. + MinP float32 `json:"min_p,omitempty"` + // TypicalP is a parameter that helps to generate more "typical" or expected text by sampling from a reduced set of tokens that are considered typical. + TypicalP float32 `json:"typical_p,omitempty"` + // RepeatLastN specifies how many of the last N tokens to consider for penalizing repetition. + RepeatLastN int `json:"repeat_last_n,omitempty"` + // Temperature controls the randomness of the generated text. A higher temperature results in more random and creative output, while a lower temperature produces more predictable and conservative text. + Temperature float32 `json:"temperature,omitempty"` + // RepeatPenalty is used to penalize the model for repeating tokens that have already appeared in the generated text. + RepeatPenalty float32 `json:"repeat_penalty,omitempty"` + // PresencePenalty is used to penalize the model for introducing new tokens that were not present in the prompt. + PresencePenalty float32 `json:"presence_penalty,omitempty"` + // FrequencyPenalty is used to penalize the model for using tokens that appear frequently in the training data. + FrequencyPenalty float32 `json:"frequency_penalty,omitempty"` + // Stop is a list of strings that will cause the generation to stop if they are encountered. + Stop []string `json:"stop,omitempty"` +} + +type ThinkValue struct { + // Value can be a bool or string + Value interface{} +} + ``` -Example of message formats: + +## examples + +### generate ```go -import "github.com/cloudwego/eino/schema" - -messages := []*schema.Message{ - // System message - schema.SystemMessage("You are an assistant"), - - // User message - schema.UserMessage("Hello") + +package main + +import ( + "context" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ollama" +) + +func main() { + ctx := context.Background() + modelName := os.Getenv("MODEL_NAME") + chatModel, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ + BaseURL: "http://localhost:11434", + Model: modelName, + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v\n", err) + return + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "as a machine, how do you answer user's question?", + }, + }) + if err != nil { + log.Printf("Generate failed, err=%v\n", err) + return + } + + log.Printf("output: \n%v\n", resp) } + ``` -### **Tool Invocation** +### generate_with_image -Supports binding tools: +```go -> Note: Only models that support function call can use this capability +package main -```go -import "github.com/cloudwego/eino/schema" - -// Define tools -tools := []*schema.ToolInfo{ - { - Name: "search", - Desc: "Search information", - ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{ - "query": { - Type: schema.String, - Desc: "Search keyword", - Required: true, - }, - }), - }, +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ollama" +) + +func main() { + ctx := context.Background() + modelName := os.Getenv("MODEL_NAME") + chatModel, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ + BaseURL: "http://localhost:11434", + Model: modelName, + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v\n", err) + return + } + + multiModalMsg := &schema.Message{ + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "this picture is a landscape photo, what's the picture's content", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + URL: of("https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcT11qEDxU4X_MVKYQVU5qiAVFidA58f8GG0bQ&s"), + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + }, + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + multiModalMsg, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + fmt.Printf("output: \n%v", resp) } - -// Bind tools -err := model.BindTools(tools) + +func of[T any](a T) *T { + return &a +} + ``` -### **Complete Usage Example** +### stream + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ollama" +) + +func main() { + ctx := context.Background() + modelName := os.Getenv("MODEL_NAME") + chatModel, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ + BaseURL: "http://localhost:11434", + Model: modelName, + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v", err) + return + } + + streamMsgs, err := chatModel.Stream(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "as a machine, how do you answer user's question?", + }, + }) + + if err != nil { + log.Printf("Generate failed, err=%v", err) + return + } + + defer streamMsgs.Close() + + log.Println("typewriter output:") + for { + msg, err := streamMsgs.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Printf("\nstream.Recv failed, err=%v", err) + return + } + fmt.Print(msg.Content) + } + + fmt.Print("\n") +} -#### **Basic Conversation** +``` + +### intent_tool ```go + package main import ( - "context" - "time" + "context" + "log" + "os" - "github.com/cloudwego/eino-ext/components/model/ollama" - "github.com/cloudwego/eino/schema" - "github.com/ollama/ollama/api" + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ollama" ) func main() { - ctx := context.Background() - - // Initialize model - model, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ - BaseURL: "http://localhost:11434", - Timeout: 30 * time.Second, - Model: "llama2", - Options: &api.Options{ - Temperature: 0.7, - NumPredict: 100, - }, - }) - if err != nil { - panic(err) - } - - // Prepare messages - messages := []*schema.Message{ - schema.SystemMessage("You are an assistant"), - schema.UserMessage("Introduce Ollama"), - } - - // Generate response - response, err := model.Generate(ctx, messages) - if err != nil { - panic(err) - } - - // Handle response - println(response.Content) + + ctx := context.Background() + modelName := os.Getenv("MODEL_NAME") + chatModel, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ + BaseURL: "http://localhost:11434", + Model: modelName, + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v", err) + return + } + + err = chatModel.BindTools([]*schema.ToolInfo{ + { + Name: "user_company", + Desc: "Query the user's company and position information based on their name and email", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "The user's name", + }, + "email": { + Type: "string", + Desc: "The user's email", + }, + }), + }, + { + Name: "user_salary", + Desc: "Query the user's salary information based on their name and email", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "The user's name", + }, + "email": { + Type: "string", + Desc: "The user's email", + }, + }), + }, + }) + if err != nil { + log.Printf("BindForcedTools failed, err=%v", err) + return + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "You are a real estate agent. Use the user_company and user_salary APIs to provide relevant property information based on the user's salary and job. Email is required", + }, + { + Role: schema.User, + Content: "My name is zhangsan, and my email is zhangsan@bytedance.com. Please recommend some suitable houses for me.", + }, + }) + + if err != nil { + log.Printf("Generate failed, err=%v", err) + return + } + + log.Printf("output: \n%+v", resp) } + ``` -#### **Streaming Conversation** +### thinking ```go + package main import ( - "context" - "time" - - "github.com/cloudwego/eino-ext/components/model/ollama" - "github.com/cloudwego/eino/schema" + "context" + "log" + "os" + + "github.com/cloudwego/eino/schema" + ollamaapi "github.com/eino-contrib/ollama/api" + + "github.com/cloudwego/eino-ext/components/model/ollama" ) func main() { - ctx := context.Background() - - // Initialize model - model, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ - BaseURL: "http://localhost:11434", - Timeout: 30 * time.Second, - Model: "llama2", - }) - if err != nil { - panic(err) - } - - // Prepare messages - messages := []*schema.Message{ - schema.SystemMessage("You are an assistant"), - schema.UserMessage("Tell a joke"), - } - - // Get streaming response - stream, err := model.Stream(ctx, messages) - if err != nil { - panic(err) - } - defer stream.Close() // Remember to close the reader - - // Handle streaming content - for { - chunk, err := stream.Recv() - if err != nil { - break - } - print(chunk.Content) - } + ctx := context.Background() + modelName := os.Getenv("MODEL_NAME") + thinking := ollamaapi.ThinkValue{Value: true} + chatModel, err := ollama.NewChatModel(ctx, &ollama.ChatModelConfig{ + BaseURL: "http://localhost:11434", + Model: modelName, + Thinking: &thinking, + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v\n", err) + return + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "as a machine, how do you answer user's question?", + }, + }) + if err != nil { + log.Printf("Generate failed, err=%v\n", err) + return + } + + log.Printf("output thinking: \n%v\n", resp.ReasoningContent) + log.Printf("output content: \n%v\n", resp.Content) } + ``` -## **Related Documents** -- [Eino: ChatModel guide](/docs/eino/core_modules/components/chat_model_guide) -- [Eino: ToolsNode guide](/docs/eino/core_modules/components/tools_node_guide) -- [ChatModel - OpenAI](/docs/eino/ecosystem_integration/chat_model/chat_model_openai) -- [ChatModel - ARK](/docs/eino/ecosystem_integration/chat_model/chat_model_ark) -- [Ollama Model Library](https://ollama.ai/library) + +## For More Details + +- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) +- [Ollama Documentation](https://ollama.readthedocs.io/api/#generate-a-chat-completion) diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_openai.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_openai.md index aa364f3e440..648d849a0bb 100644 --- a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_openai.md +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_openai.md @@ -1,229 +1,668 @@ --- Description: "" -date: "2025-03-19" +date: "2025-12-02" lastmod: "" tags: [] -title: ChatModel - OpenAI +title: ChatModel - openai weight: 0 --- -## **Basic Introduction** +A OpenAI model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. -The OpenAI model is an implementation of the ChatModel interface, used for interacting with OpenAI's GPT series models. This component implements the [Eino: ChatModel guide](/docs/eino/core_modules/components/chat_model_guide), primarily for the following scenarios: +## Features -- Need to use OpenAI's GPT series models -- Need to use Azure OpenAI Service -- Use other models compatible with OpenAI interfaces +- Implements `github.com/cloudwego/eino/components/model.Model` +- Easy integration with Eino's model system +- Configurable model parameters +- Support for chat completion +- Support for streaming responses +- Custom response parsing support +- Flexible model configuration -## **Usage Instructions** +## Installation -### **Component Initialization** +```bash +go get github.com/cloudwego/eino-ext/components/model/openai@latest +``` + +## Quick Start -The OpenAI model is initialized via the `NewChatModel` function. The main configuration parameters are as follows: +Here's a quick example of how to use the OpenAI model: ```go -import "github.com/cloudwego/eino-ext/components/model/openai" - -model, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ - // Azure OpenAI Service configuration (optional) - ByAzure: false, // Whether to use Azure OpenAI - BaseURL: "your-url", // Azure API base URL - APIVersion: "2023-05-15", // Azure API version - - // Basic configuration - APIKey: "your-key", // API key - Timeout: 30 * time.Second, // Timeout duration - - // Model parameters - Model: "gpt-4", // Model name - MaxTokens: &maxTokens,// Maximum generation length - Temperature: &temp, // Temperature - TopP: &topP, // Top-P sampling - Stop: []string{},// Stop words - PresencePenalty: &pp, // Presence penalty - FrequencyPenalty: &fp, // Frequency penalty - - // Advanced parameters - ResponseFormat: &format, // Response format - Seed: &seed, // Random seed - LogitBias: map[string]int{}, // Token bias - User: &user, // User identifier -}) -``` -> - For detailed parameter meanings, refer to: [https://platform.openai.com/docs/api-reference/chat/create](https://platform.openai.com/docs/api-reference/chat/create) -> - For Azure related services, refer to: [https://learn.microsoft.com/en-us/azure/ai-services/openai/](https://learn.microsoft.com/en-us/azure/ai-services/openai/) +package main -### **Generating Dialogues** +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/openai" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + + chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ + // if you want to use Azure OpenAI Service, set these two field. + // BaseURL: "https://{RESOURCE_NAME}.openai.azure.com", + // ByAzure: true, + // APIVersion: "2024-06-01", + APIKey: os.Getenv("OPENAI_API_KEY"), + Model: os.Getenv("OPENAI_MODEL"), + BaseURL: os.Getenv("OPENAI_BASE_URL"), + ByAzure: func() bool { + if os.Getenv("OPENAI_BY_AZURE") == "true" { + return true + } + return false + }(), + ReasoningEffort: openai.ReasoningEffortLevelHigh, + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "as a machine, how do you answer user's question?", + }, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + fmt.Printf("output: \n%v", resp) + +} -Dialogue generation supports both regular mode and streaming mode: -```go -// invoke mode -response, err := model.Generate(ctx, messages) - -// streaming mode -stream, err := model.Stream(ctx, messages) ``` -Example of message format: +## Configuration + +The model can be configured using the `openai.ChatModelConfig` struct: ```go -import "github.com/cloudwego/eino/schema" - -messages := []*schema.Message{ - // System message - schema.SystemMessage("You are an assistant"), - - // Multi-modal message (containing an image) - { - Role: schema.User, - MultiContent: []schema.ChatMessagePart{ - { - Type: schema.ChatMessagePartTypeImageURL, - ImageURL: &schema.ChatMessageImageURL{ - URL: "https://example.com/image.jpg", - Detail: "high", - }, - }, - { - Type: schema.ChatMessagePartTypeText, - Text: "What is in this image?", - }, - }, - }, + +type ChatModelConfig struct { +// APIKey is your authentication key +// Use OpenAI API key or Azure API key depending on the service +// Required +APIKey string `json:"api_key"` + +// Timeout specifies the maximum duration to wait for API responses +// If HTTPClient is set, Timeout will not be used. +// Optional. Default: no timeout +Timeout time.Duration `json:"timeout"` + +// HTTPClient specifies the client to send HTTP requests. +// If HTTPClient is set, Timeout will not be used. +// Optional. Default &http.Client{Timeout: Timeout} +HTTPClient *http.Client `json:"http_client"` + +// The following three fields are only required when using Azure OpenAI Service, otherwise they can be ignored. +// For more details, see: https://learn.microsoft.com/en-us/azure/ai-services/openai/ + +// ByAzure indicates whether to use Azure OpenAI Service +// Required for Azure +ByAzure bool `json:"by_azure"` + +// AzureModelMapperFunc is used to map the model name to the deployment name for Azure OpenAI Service. +// This is useful when the model name is different from the deployment name. +// Optional for Azure, remove [,:] from the model name by default. +AzureModelMapperFunc func(model string) string + +// BaseURL is the Azure OpenAI endpoint URL +// Format: https://{YOUR_RESOURCE_NAME}.openai.azure.com. YOUR_RESOURCE_NAME is the name of your resource that you have created on Azure. +// Required for Azure +BaseURL string `json:"base_url"` + +// APIVersion specifies the Azure OpenAI API version +// Required for Azure +APIVersion string `json:"api_version"` + +// The following fields correspond to OpenAI's chat completion API parameters +// Ref: https://platform.openai.com/docs/api-reference/chat/create + +// Model specifies the ID of the model to use +// Required +Model string `json:"model"` + +// MaxTokens limits the maximum number of tokens that can be generated in the chat completion +// Optional. Default: model's maximum +// Deprecated: use MaxCompletionTokens. Not compatible with o1-series models. +// refs: https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_tokens +MaxTokens *int `json:"max_tokens,omitempty"` + +// MaxCompletionTokens specifies an upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens. +MaxCompletionTokens *int `json:"max_completion_tokens,omitempty"` + +// Temperature specifies what sampling temperature to use +// Generally recommend altering this or TopP but not both. +// Range: 0.0 to 2.0. Higher values make output more random +// Optional. Default: 1.0 +Temperature *float32 `json:"temperature,omitempty"` + +// TopP controls diversity via nucleus sampling +// Generally recommend altering this or Temperature but not both. +// Range: 0.0 to 1.0. Lower values make output more focused +// Optional. Default: 1.0 +TopP *float32 `json:"top_p,omitempty"` + +// Stop sequences where the API will stop generating further tokens +// Optional. Example: []string{"\n", "User:"} +Stop []string `json:"stop,omitempty"` + +// PresencePenalty prevents repetition by penalizing tokens based on presence +// Range: -2.0 to 2.0. Positive values increase likelihood of new topics +// Optional. Default: 0 +PresencePenalty *float32 `json:"presence_penalty,omitempty"` + +// ResponseFormat specifies the format of the model's response +// Optional. Use for structured outputs +ResponseFormat *ChatCompletionResponseFormat `json:"response_format,omitempty"` + +// Seed enables deterministic sampling for consistent outputs +// Optional. Set for reproducible results +Seed *int `json:"seed,omitempty"` + +// FrequencyPenalty prevents repetition by penalizing tokens based on frequency +// Range: -2.0 to 2.0. Positive values decrease likelihood of repetition +// Optional. Default: 0 +FrequencyPenalty *float32 `json:"frequency_penalty,omitempty"` + +// LogitBias modifies likelihood of specific tokens appearing in completion +// Optional. Map token IDs to bias values from -100 to 100 +LogitBias map[string]int `json:"logit_bias,omitempty"` + +// User unique identifier representing end-user +// Optional. Helps OpenAI monitor and detect abuse +User *string `json:"user,omitempty"` + +// ExtraFields will override any existing fields with the same key. +// Optional. Useful for experimental features not yet officially supported. +ExtraFields map[string]any `json:"extra_fields,omitempty"` + +// ReasoningEffort will override the default reasoning level of "medium" +// Optional. Useful for fine tuning response latency vs. accuracy +ReasoningEffort ReasoningEffortLevel + +// Modalities are output types that you would like the model to generate. Most models are capable of generating text, which is the default: ["text"] +// The gpt-4o-audio-preview model can also be used to generate audio. To request that this model generate both text and audio responses, you can use: ["text", "audio"] +Modalities []Modality `json:"modalities,omitempty"` + +// Audio parameters for audio output. Required when audio output is requested with modalities: ["audio"] +Audio *Audio `json:"audio,omitempty"` } ``` -### **Tool Invocation** -Supports binding tools and forced tool invocation: +## examples + +### generate ```go -import "github.com/cloudwego/eino/schema" - -// Define tools -tools := []*schema.ToolInfo{ - { - Name: "search", - Desc: "Search information", - ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{ - "query": { - Type: schema.String, - Desc: "Search keywords", - Required: true, - }, - }), - }, + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/openai" +) + +func main() { + ctx := context.Background() + + chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ + // if you want to use Azure OpenAI Service, set these two field. + // BaseURL: "https://{RESOURCE_NAME}.openai.azure.com", + // ByAzure: true, + // APIVersion: "2024-06-01", + APIKey: os.Getenv("OPENAI_API_KEY"), + Model: os.Getenv("OPENAI_MODEL"), + BaseURL: os.Getenv("OPENAI_BASE_URL"), + ByAzure: func() bool { + if os.Getenv("OPENAI_BY_AZURE") == "true" { + return true + } + return false + }(), + ReasoningEffort: openai.ReasoningEffortLevelHigh, + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "as a machine, how do you answer user's question?", + }, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + fmt.Printf("output: \n%v", resp) + +} + +``` + +### generate_with_image + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/openai" +) + +func main() { + ctx := context.Background() + + chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ + APIKey: os.Getenv("OPENAI_API_KEY"), + Model: os.Getenv("OPENAI_MODEL"), + BaseURL: os.Getenv("OPENAI_BASE_URL"), + ByAzure: func() bool { + if os.Getenv("OPENAI_BY_AZURE") == "true" { + return true + } + return false + }(), + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + + } + + multiModalMsg := &schema.Message{ + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "this picture is a landscape photo, what's the picture's content", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + URL: of("https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcT11qEDxU4X_MVKYQVU5qiAVFidA58f8GG0bQ&s"), + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + }, + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + multiModalMsg, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + fmt.Printf("output: \n%v", resp) } -// Bind optional tools -err := model.BindTools(tools) +func of[T any](a T) *T { + return &a +} -// Bind forced tools -err := model.BindForcedTools(tools) ``` -> For tool-related information, refer to [Eino: ToolsNode guide](/docs/eino/core_modules/components/tools_node_guide) +### stream -### **Complete Usage Example** +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/openai" +) + +func main() { + ctx := context.Background() + chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ + APIKey: os.Getenv("OPENAI_API_KEY"), + Model: os.Getenv("OPENAI_MODEL"), + BaseURL: os.Getenv("OPENAI_BASE_URL"), + ByAzure: func() bool { + if os.Getenv("OPENAI_BY_AZURE") == "true" { + return true + } + return false + }(), + }) + if err != nil { + log.Fatalf("NewChatModel of openai failed, err=%v", err) + } + + streamMsgs, err := chatModel.Stream(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "as a machine, how do you answer user's question?", + }, + }) + + if err != nil { + log.Fatalf("Stream of openai failed, err=%v", err) + } + + defer streamMsgs.Close() + + fmt.Printf("typewriter output:") + for { + msg, err := streamMsgs.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Fatalf("Recv of streamMsgs failed, err=%v", err) + } + fmt.Print(msg.Content) + } + + fmt.Print("\n") +} + +``` + +### intent_tool + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" -#### **Direct Conversation** + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/openai" +) + +func main() { + ctx := context.Background() + chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ + APIKey: os.Getenv("OPENAI_API_KEY"), + Model: os.Getenv("OPENAI_MODEL"), + BaseURL: os.Getenv("OPENAI_BASE_URL"), + ByAzure: func() bool { + if os.Getenv("OPENAI_BY_AZURE") == "true" { + return true + } + return false + }(), + }) + if err != nil { + log.Fatalf("NewChatModel of openai failed, err=%v", err) + } + err = chatModel.BindForcedTools([]*schema.ToolInfo{ + { + Name: "user_company", + Desc: "Retrieve the user's company and position based on their name and email.", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": {Type: "string", Desc: "user's name"}, + "email": {Type: "string", Desc: "user's email"}}), + }, { + Name: "user_salary", + Desc: "Retrieve the user's salary based on their name and email.\n", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": {Type: "string", Desc: "user's name"}, + "email": {Type: "string", Desc: "user's email"}, + }), + }}) + if err != nil { + log.Fatalf("BindForcedTools of openai failed, err=%v", err) + } + resp, err := chatModel.Generate(ctx, []*schema.Message{{ + Role: schema.System, + Content: "As a real estate agent, provide relevant property information based on the user's salary and job using the user_company and user_salary APIs. An email address is required.", + }, { + Role: schema.User, + Content: "My name is John and my email is john@abc.com，Please recommend some houses that suit me.", + }}) + if err != nil { + log.Fatalf("Generate of openai failed, err=%v", err) + } + fmt.Printf("output: \n%v", resp) + + streamResp, err := chatModel.Stream(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "As a real estate agent, provide relevant property information based on the user's salary and job using the user_company and user_salary APIs. An email address is required.", + }, { + Role: schema.User, + Content: "My name is John and my email is john@abc.com，Please recommend some houses that suit me.", + }, + }) + if err != nil { + log.Fatalf("Stream of openai failed, err=%v", err) + } + var messages []*schema.Message + for { + chunk, err := streamResp.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Fatalf("Recv of streamResp failed, err=%v", err) + } + messages = append(messages, chunk) + } + resp, err = schema.ConcatMessages(messages) + if err != nil { + log.Fatalf("ConcatMessages of openai failed, err=%v", err) + } + fmt.Printf("stream output: \n%v", resp) +} + +``` + +### audio_generate ```go + package main import ( - "context" - "time" - - "github.com/cloudwego/eino-ext/components/model/openai" - "github.com/cloudwego/eino/schema" + "context" + + "log" + "os" + + "github.com/bytedance/sonic" + "github.com/cloudwego/eino-ext/components/model/openai" + "github.com/cloudwego/eino/schema" ) func main() { - ctx := context.Background() - - // Initialize model - model, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ - APIKey: "your-api-key", // required - Timeout: 30 * time.Second, - Model: "gpt-4", // required - }) - if err != nil { - panic(err) - } - - // Prepare messages - messages := []*schema.Message{ - schema.SystemMessage("You are an assistant"), - schema.UserMessage("Introduce eino"), - } - - // Generate response - response, err := model.Generate(ctx, messages) - if err != nil { - panic(err) - } - - // Handle response - println(response.Content) + ctx := context.Background() + + chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ + // if you want to use Azure OpenAI Service, set these two field. + // BaseURL: "https://{RESOURCE_NAME}.openai.azure.com", + // ByAzure: true, + // APIVersion: "2024-06-01", + APIKey: os.Getenv("OPENAI_API_KEY"), + Model: os.Getenv("OPENAI_MODEL"), + BaseURL: os.Getenv("OPENAI_BASE_URL"), + ByAzure: func() bool { + if os.Getenv("OPENAI_BY_AZURE") == "true" { + return true + } + return false + }(), + ReasoningEffort: openai.ReasoningEffortLevelHigh, + Modalities: []openai.Modality{openai.AudioModality, openai.TextModality}, + Audio: &openai.Audio{ + Format: openai.AudioFormatMp3, + Voice: openai.AudioVoiceAlloy, + }, + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + {Type: schema.ChatMessagePartTypeText, Text: "help me convert the following text to speech"}, + {Type: schema.ChatMessagePartTypeText, Text: "Hello, what can I help you with?"}, + }, + }, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + respBody, _ := sonic.MarshalIndent(resp, " ", " ") + log.Printf(" body: %s\n", string(respBody)) + } + ``` -#### **Streaming Conversation** +### structured ```go + package main import ( - "context" - "time" - - "github.com/cloudwego/eino-ext/components/model/openai" - "github.com/cloudwego/eino/schema" + "context" + "encoding/json" + "fmt" + "log" + "os" + + "github.com/eino-contrib/jsonschema" + orderedmap "github.com/wk8/go-ordered-map/v2" + + "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/openai" ) func main() { - ctx := context.Background() - - // Initialize model - model, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ - APIKey: "your-api-key", - Timeout: 30 * time.Second, - Model: "gpt-4", - }) - if err != nil { - panic(err) - } - - // Prepare messages - messages := []*schema.Message{ - schema.SystemMessage("You are an assistant"), - schema.UserMessage("Write a story"), - } - - // Get streaming response - reader, err := model.Stream(ctx, messages) - if err != nil { - panic(err) - } - defer reader.Close() // Remember to close - - // Handle streaming content - for { - chunk, err := reader.Recv() - if err != nil { - break - } - print(chunk.Content) - } + type Person struct { + Name string `json:"name"` + Height int `json:"height"` + Weight int `json:"weight"` + } + + js := &jsonschema.Schema{ + Type: string(schema.Object), + Properties: orderedmap.New[string, *jsonschema.Schema]( + orderedmap.WithInitialData[string, *jsonschema.Schema]( + orderedmap.Pair[string, *jsonschema.Schema]{ + Key: "name", + Value: &jsonschema.Schema{ + Type: string(schema.String), + }, + }, + orderedmap.Pair[string, *jsonschema.Schema]{ + Key: "height", + Value: &jsonschema.Schema{ + Type: string(schema.Integer), + }, + }, + orderedmap.Pair[string, *jsonschema.Schema]{ + Key: "weight", + Value: &jsonschema.Schema{ + Type: string(schema.Integer), + }, + }, + ), + ), + } + + ctx := context.Background() + chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ + APIKey: os.Getenv("OPENAI_API_KEY"), + Model: os.Getenv("OPENAI_MODEL"), + BaseURL: os.Getenv("OPENAI_BASE_URL"), + ByAzure: func() bool { + if os.Getenv("OPENAI_BY_AZURE") == "true" { + return true + } + return false + }(), + ResponseFormat: &openai.ChatCompletionResponseFormat{ + Type: openai.ChatCompletionResponseFormatTypeJSONSchema, + JSONSchema: &openai.ChatCompletionResponseFormatJSONSchema{ + Name: "person", + Description: "data that describes a person", + Strict: false, + JSONSchema: js, + }, + }, + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "Parse the user input into the specified json struct", + }, + { + Role: schema.User, + Content: "John is one meter seventy tall and weighs sixty kilograms", + }, + }) + + if err != nil { + log.Fatalf("Generate of openai failed, err=%v", err) + } + + result := &Person{} + err = json.Unmarshal([]byte(resp.Content), result) + if err != nil { + log.Fatalf("Unmarshal of openai failed, err=%v", err) + } + fmt.Printf("%+v", *result) } + ``` -## **Related Documents** -- [Eino: ChatModel guide](/docs/eino/core_modules/components/chat_model_guide) -- [Eino: ToolsNode guide](/docs/eino/core_modules/components/tools_node_guide) -- [ChatModel - ARK](/docs/eino/ecosystem_integration/chat_model/chat_model_ark) -- [ChatModel - Ollama](/docs/eino/ecosystem_integration/chat_model/chat_model_ollama) + +## For More Details + +- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) +- [OpenAI Documentation](https://platform.openai.com/docs/api-reference/chat/create) diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_qianfan.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_qianfan.md index 9b45043c25b..5c742d940a6 100644 --- a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_qianfan.md +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_qianfan.md @@ -1,10 +1,446 @@ --- Description: "" -date: "2025-03-19" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - qianfan weight: 0 --- -you can use LLM served by qianfan from baidu. +A Qianfan model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. + +## Features + +- Implements `github.com/cloudwego/eino/components/model.Model` +- Easy integration with Eino's model system +- Configurable model parameters +- Support for chat completion +- Support for streaming responses +- Custom response parsing support +- Flexible model configuration + +## Installation + +```bash +go get github.com/cloudwego/eino-ext/components/model/qianfan@latest +``` + +## Quick Start + +Here's a quick example of how to use the Qianfan model: + + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qianfan" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + qcfg := qianfan.GetQianfanSingletonConfig() + // How to get Access Key/Secret Key: https://cloud.baidu.com/doc/Reference/s/9jwvz2egb + qcfg.AccessKey = "your_access_key" + qcfg.SecretKey = "your_secret_key" + modelName := os.Getenv("MODEL_NAME") + cm, err := qianfan.NewChatModel(ctx, &qianfan.ChatModelConfig{ + Model: modelName, + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + MaxCompletionTokens: of(1024), + Seed: of(0), + }) + + if err != nil { + log.Fatalf("NewChatModel of qianfan failed, err=%v", err) + } + + ir, err := cm.Generate(ctx, []*schema.Message{ + schema.UserMessage("hello"), + }) + + if err != nil { + log.Fatalf("Generate of qianfan failed, err=%v", err) + } + + fmt.Println(ir) + // assistant: 你好！我是文心一言，很高兴与你交流。请问你有什么想问我的吗？无论是关于知识、创作还是其他任何问题，我都会尽力回答你。 +} + +func of[T any](t T) *T { + return &t +} +``` + +## Configuration + +The model can be configured using the `qianfan.ChatModelConfig` struct: + + +```go +// ChatModelConfig config for qianfan chat completion +// see: https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Wm3fhy2vb +type ChatModelConfig struct { + // Model is the model to use for the chat completion. + Model string + + // LLMRetryCount is the number of times to retry a failed request. + LLMRetryCount *int + + // LLMRetryTimeout is the timeout for each retry attempt. + LLMRetryTimeout *float32 + + // LLMRetryBackoffFactor is the backoff factor for retries. + LLMRetryBackoffFactor *float32 + + // Temperature controls the randomness of the output. A higher value makes the output more random, while a lower value makes it more focused and deterministic. Default is 0.95, range (0, 1.0]. + Temperature *float32 + + // TopP controls the diversity of the output. A higher value increases the diversity of the generated text. Default is 0.7, range [0, 1.0]. + TopP *float32 + + // PenaltyScore reduces the generation of repetitive tokens by adding a penalty. A higher value means a larger penalty. Range: [1.0, 2.0]. + PenaltyScore *float64 + + // MaxCompletionTokens is the maximum number of tokens to generate in the completion. Range [2, 2048]. + MaxCompletionTokens *int + + // Seed is the random seed for generation. Range (0, 2147483647). + Seed *int + + // Stop is a list of strings that will stop the generation when the model generates a token that is a suffix of one of the strings. + Stop []string + + // User is a unique identifier representing the end-user. + User *string + + // FrequencyPenalty specifies the frequency penalty to control the repetition of generated text. Range [-2.0, 2.0]. + FrequencyPenalty *float64 + + // PresencePenalty specifies the presence penalty to control the repetition of generated text. Range [-2.0, 2.0]. + PresencePenalty *float64 + + // ParallelToolCalls specifies whether to call tools in parallel. Defaults to true. + ParallelToolCalls *bool + + // ResponseFormat specifies the format of the response. + ResponseFormat *ResponseFormat +} + +``` + + + + + + + + +## examples + +### generate + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qianfan" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + + qcfg := qianfan.GetQianfanSingletonConfig() + // How to get Access Key/Secret Key: https://cloud.baidu.com/doc/Reference/s/9jwvz2egb + qcfg.AccessKey = "your_access_key" + qcfg.SecretKey = "your_secret_key" + modelName := os.Getenv("MODEL_NAME") + cm, err := qianfan.NewChatModel(ctx, &qianfan.ChatModelConfig{ + Model: modelName, + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + MaxCompletionTokens: of(1024), + Seed: of(0), + }) + + if err != nil { + log.Fatalf("NewChatModel of qianfan failed, err=%v", err) + } + + ir, err := cm.Generate(ctx, []*schema.Message{ + schema.UserMessage("hello"), + }) + + if err != nil { + log.Fatalf("Generate of qianfan failed, err=%v", err) + } + + fmt.Println(ir) +} + +func of[T any](t T) *T { + return &t +} + +``` + +### generate_with_image + +```go + +package main + +import ( + "context" + "encoding/base64" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qianfan" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + qcfg := qianfan.GetQianfanSingletonConfig() + // How to get Access Key/Secret Key: https://cloud.baidu.com/doc/Reference/s/9jwvz2egb + qcfg.AccessKey = "your_access_key" + qcfg.SecretKey = "your_secret_key" + modelName := os.Getenv("MODEL_NAME") + chatModel, err := qianfan.NewChatModel(ctx, &qianfan.ChatModelConfig{ + Model: modelName, + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + MaxCompletionTokens: of(1024), + Seed: of(0), + }) + + if err != nil { + log.Fatalf("NewChatModel of qianfan failed, err=%v", err) + } + + image, err := os.ReadFile("./examples/generate_with_image/test.jpg") + if err != nil { + log.Fatalf("os.ReadFile failed, err=%v\n", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "What do you see in this image?", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: of(base64.StdEncoding.EncodeToString(image)), + MIMEType: "image/jpeg", + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + }, + }, + }) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + fmt.Printf("Assistant: %s\n", resp.Content) +} + +func of[T any](t T) *T { + return &t +} + +``` + +### stream + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qianfan" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + qcfg := qianfan.GetQianfanSingletonConfig() + // How to get Access Key/Secret Key: https://cloud.baidu.com/doc/Reference/s/9jwvz2egb + qcfg.AccessKey = "your_access_key" + qcfg.SecretKey = "your_secret_key" + modelName := os.Getenv("MODEL_NAME") + cm, err := qianfan.NewChatModel(ctx, &qianfan.ChatModelConfig{ + Model: modelName, + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + MaxCompletionTokens: of(1024), + }) + if err != nil { + log.Fatalf("NewChatModel of qianfan failed, err=%v", err) + } + + sr, err := cm.Stream(ctx, []*schema.Message{ + schema.UserMessage("hello"), + }) + + if err != nil { + log.Fatalf("Stream of qianfan failed, err=%v", err) + } + + var ms []*schema.Message + for { + m, err := sr.Recv() + if err != nil { + if err == io.EOF { + break + } + + log.Fatalf("Stream of qianfan failed, err=%v", err) + } + + fmt.Println(m) + ms = append(ms, m) + } + sm, err := schema.ConcatMessages(ms) + if err != nil { + log.Fatalf("ConcatMessages failed, err=%v", err) + } + + fmt.Println(sm) +} + +func of[T any](t T) *T { + return &t +} + +``` + +### tool + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qianfan" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + qcfg := qianfan.GetQianfanSingletonConfig() + // How to get Access Key/Secret Key: https://cloud.baidu.com/doc/Reference/s/9jwvz2egb + qcfg.AccessKey = "your_access_key" + qcfg.SecretKey = "your_secret_key" + + modelName := os.Getenv("MODEL_NAME") + cm, err := qianfan.NewChatModel(ctx, &qianfan.ChatModelConfig{ + Model: modelName, + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + MaxCompletionTokens: of(1024), + }) + if err != nil { + log.Fatalf("NewChatModel of qianfan failed, err=%v", err) + } + + err = cm.BindTools([]*schema.ToolInfo{ + { + Name: "user_company", + Desc: "Query the user's company and position information based on their name and email", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "The user's name", + }, + "email": { + Type: "string", + Desc: "The user's email", + }, + }), + }, + { + Name: "user_salary", + Desc: "Query the user's salary information based on their name and email", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "The user's name", + }, + "email": { + Type: "string", + Desc: "The user's email", + }, + }), + }, + }) + if err != nil { + log.Fatalf("BindTools of qianfan failed, err=%v", err) + } + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "You are a real estate agent. Use the user_company and user_salary APIs to provide relevant property information based on the user's salary and job. Email is required", + }, + { + Role: schema.User, + Content: "My name is zhangsan, and my email is zhangsan@bytedance.com. Please recommend some suitable houses for me.", + }, + }) + + if err != nil { + log.Fatalf("Generate of qianfan failed, err=%v", err) + } + + fmt.Println(resp) +} + +func of[T any](t T) *T { + return &t +} + +``` + + + +## For More Details + +- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) +- [Qianfan Documentation](https://cloud.baidu.com/doc/qianfan-api/s/rm7u7qdiq) diff --git a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_qwen.md b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_qwen.md index 0f8a02d6502..e01cab8ca22 100644 --- a/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_qwen.md +++ b/content/en/docs/eino/ecosystem_integration/chat_model/chat_model_qwen.md @@ -1,10 +1,532 @@ --- Description: "" -date: "2025-03-19" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - qwen weight: 0 --- -you can use qwen llm. +A Qwen model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. + +## Features + +- Implements `github.com/cloudwego/eino/components/model.Model` +- Easy integration with Eino's model system +- Configurable model parameters +- Support for chat completion +- Support for streaming responses +- Custom response parsing support +- Flexible model configuration + +## Installation + +```bash +go get github.com/cloudwego/eino-ext/components/model/qwen@latest +``` + +## Quick Start + +Here's a quick example of how to use the Qwen model: + +```go +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qwen" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + // get api key: https://help.aliyun.com/zh/model-studio/developer-reference/get-api-key?spm=a2c4g.11186623.help-menu-2400256.d_3_0.1ebc47bb0ClCgF + apiKey := os.Getenv("DASHSCOPE_API_KEY") + modelName := os.Getenv("MODEL_NAME") + chatModel, err := qwen.NewChatModel(ctx, &qwen.ChatModelConfig{ + BaseURL: "https://dashscope.aliyuncs.com/compatible-mode/v1", + APIKey: apiKey, + Timeout: 0, + Model: modelName, + MaxTokens: of(2048), + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + }) + + if err != nil { + log.Fatalf("NewChatModel of qwen failed, err=%v", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + schema.UserMessage("as a machine, how do you answer user's question?"), + }) + if err != nil { + log.Fatalf("Generate of qwen failed, err=%v", err) + } + + fmt.Printf("output: \n%v", resp) + +} + +func of[T any](t T) *T { + return &t +} +``` + +## Configuration + +The model can be configured using the `qwen.ChatModelConfig` struct: + +```go +type ChatModelConfig struct { + +// APIKey is your authentication key +// Use OpenAI API key or Azure API key depending on the service +// Required +APIKey string `json:"api_key"` + +// Timeout specifies the maximum duration to wait for API responses +// If HTTPClient is set, Timeout will not be used. +// Optional. Default: no timeout +Timeout time.Duration `json:"timeout"` + +// HTTPClient specifies the client to send HTTP requests. +// If HTTPClient is set, Timeout will not be used. +// Optional. Default &http.Client{Timeout: Timeout} +HTTPClient *http.Client `json:"http_client"` + +// BaseURL specifies the QWen endpoint URL +// Required. Example: https://dashscope.aliyuncs.com/compatible-mode/v1 +BaseURL string `json:"base_url"` + +// The following fields correspond to OpenAI's chat completion API parameters +// Ref: https://platform.openai.com/docs/api-reference/chat/create + +// Model specifies the ID of the model to use +// Required +Model string `json:"model"` + +// MaxTokens limits the maximum number of tokens that can be generated in the chat completion +// Optional. Default: model's maximum +MaxTokens *int `json:"max_tokens,omitempty"` + +// Temperature specifies what sampling temperature to use +// Generally recommend altering this or TopP but not both. +// Range: 0.0 to 2.0. Higher values make output more random +// Optional. Default: 1.0 +Temperature *float32 `json:"temperature,omitempty"` + +// TopP controls diversity via nucleus sampling +// Generally recommend altering this or Temperature but not both. +// Range: 0.0 to 1.0. Lower values make output more focused +// Optional. Default: 1.0 +TopP *float32 `json:"top_p,omitempty"` + +// Stop sequences where the API will stop generating further tokens +// Optional. Example: []string{"\n", "User:"} +Stop []string `json:"stop,omitempty"` + +// PresencePenalty prevents repetition by penalizing tokens based on presence +// Range: -2.0 to 2.0. Positive values increase likelihood of new topics +// Optional. Default: 0 +PresencePenalty *float32 `json:"presence_penalty,omitempty"` + +// ResponseFormat specifies the format of the model's response +// Optional. Use for structured outputs +ResponseFormat *openai.ChatCompletionResponseFormat `json:"response_format,omitempty"` + +// Seed enables deterministic sampling for consistent outputs +// Optional. Set for reproducible results +Seed *int `json:"seed,omitempty"` + +// FrequencyPenalty prevents repetition by penalizing tokens based on frequency +// Range: -2.0 to 2.0. Positive values decrease likelihood of repetition +// Optional. Default: 0 +FrequencyPenalty *float32 `json:"frequency_penalty,omitempty"` + +// LogitBias modifies likelihood of specific tokens appearing in completion +// Optional. Map token IDs to bias values from -100 to 100 +LogitBias map[string]int `json:"logit_bias,omitempty"` + +// User unique identifier representing end-user +// Optional. Helps OpenAI monitor and detect abuse +User *string `json:"user,omitempty"` + +// EnableThinking enables thinking mode +// https://help.aliyun.com/zh/model-studio/deep-thinking +// Optional. Default: base on the Model +EnableThinking *bool `json:"enable_thinking,omitempty"` +} + +``` + + + + + +## examples + +### generate + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qwen" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + // get api key: https://help.aliyun.com/zh/model-studio/developer-reference/get-api-key?spm=a2c4g.11186623.help-menu-2400256.d_3_0.1ebc47bb0ClCgF + apiKey := os.Getenv("DASHSCOPE_API_KEY") + modelName := os.Getenv("MODEL_NAME") + chatModel, err := qwen.NewChatModel(ctx, &qwen.ChatModelConfig{ + BaseURL: "https://dashscope.aliyuncs.com/compatible-mode/v1", + APIKey: apiKey, + Timeout: 0, + Model: modelName, + MaxTokens: of(2048), + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + }) + + if err != nil { + log.Fatalf("NewChatModel of qwen failed, err=%v", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + schema.UserMessage("as a machine, how do you answer user's question?"), + }) + if err != nil { + log.Fatalf("Generate of qwen failed, err=%v", err) + } + + fmt.Printf("output: \n%v", resp) + +} + +func of[T any](t T) *T { + return &t +} + +``` + +### generate_with_image + +```go + +package main + +import ( + "context" + "encoding/base64" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qwen" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + // get api key: https://help.aliyun.com/zh/model-studio/developer-reference/get-api-key?spm=a2c4g.11186623.help-menu-2400256.d_3_0.1ebc47bb0ClCgF + apiKey := os.Getenv("DASHSCOPE_API_KEY") + modelName := os.Getenv("MODEL_NAME") + chatModel, err := qwen.NewChatModel(ctx, &qwen.ChatModelConfig{ + BaseURL: "https://dashscope.aliyuncs.com/compatible-mode/v1", + APIKey: apiKey, + Timeout: 0, + Model: modelName, + MaxTokens: of(2048), + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + }) + + if err != nil { + log.Fatalf("NewChatModel of qwen failed, err=%v", err) + } + + image, err := os.ReadFile("./examples/generate_with_image/test.jpg") + if err != nil { + log.Fatalf("os.ReadFile failed, err=%v\n", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "What do you see in this image?", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: of(base64.StdEncoding.EncodeToString(image)), + MIMEType: "image/jpeg", + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + }, + }, + }) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + fmt.Printf("Assistant: %s\n", resp.Content) + +} + +func of[T any](t T) *T { + return &t +} + +``` + +### stream + +```go + +package main + +import ( + "context" + "fmt" + "io" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qwen" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + // get api key: https://help.aliyun.com/zh/model-studio/developer-reference/get-api-key?spm=a2c4g.11186623.help-menu-2400256.d_3_0.1ebc47bb0ClCgF + apiKey := os.Getenv("DASHSCOPE_API_KEY") + modelName := os.Getenv("MODEL_NAME") + cm, err := qwen.NewChatModel(ctx, &qwen.ChatModelConfig{ + BaseURL: "https://dashscope.aliyuncs.com/compatible-mode/v1", + APIKey: apiKey, + Timeout: 0, + Model: modelName, + MaxTokens: of(2048), + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + }) + if err != nil { + log.Fatalf("NewChatModel of qwen failed, err=%v", err) + } + + sr, err := cm.Stream(ctx, []*schema.Message{ + schema.UserMessage("你好"), + }) + if err != nil { + log.Fatalf("Stream of qwen failed, err=%v", err) + } + + var msgs []*schema.Message + for { + msg, err := sr.Recv() + if err != nil { + if err == io.EOF { + break + } + + log.Fatalf("Stream of qwen failed, err=%v", err) + } + + fmt.Println(msg) + // assistant: 你好 + // finish_reason: + // : ！ + // finish_reason: + // : 有什么 + // finish_reason: + // : 可以帮助 + // finish_reason: + // : 你的吗？ + // finish_reason: + // : + // finish_reason: stop + // usage: &{9 7 16} + msgs = append(msgs, msg) + } + + msg, err := schema.ConcatMessages(msgs) + if err != nil { + log.Fatalf("ConcatMessages failed, err=%v", err) + } + + fmt.Println(msg) + // assistant: 你好！有什么可以帮助你的吗？ + // finish_reason: stop + // usage: &{9 7 16} +} + +func of[T any](t T) *T { + return &t +} + +``` + +### tool + +```go + +package main + +import ( + "context" + "encoding/json" + "fmt" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/qwen" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + // get api key: https://help.aliyun.com/zh/model-studio/developer-reference/get-api-key?spm=a2c4g.11186623.help-menu-2400256.d_3_0.1ebc47bb0ClCgF + apiKey := os.Getenv("DASHSCOPE_API_KEY") + modelName := os.Getenv("MODEL_NAME") + cm, err := qwen.NewChatModel(ctx, &qwen.ChatModelConfig{ + BaseURL: "https://dashscope.aliyuncs.com/compatible-mode/v1", + APIKey: apiKey, + Timeout: 0, + Model: modelName, + MaxTokens: of(2048), + Temperature: of(float32(0.7)), + TopP: of(float32(0.7)), + }) + if err != nil { + log.Fatalf("NewChatModel of qwen failed, err=%v", err) + } + + err = cm.BindTools([]*schema.ToolInfo{ + { + Name: "user_company", + Desc: "根据用户的姓名和邮箱，查询用户的公司和职位信息", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "用户的姓名", + }, + "email": { + Type: "string", + Desc: "用户的邮箱", + }, + }), + }, + { + Name: "user_salary", + Desc: "根据用户的姓名和邮箱，查询用户的薪酬信息", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "用户的姓名", + }, + "email": { + Type: "string", + Desc: "用户的邮箱", + }, + }), + }, + }) + if err != nil { + log.Fatalf("BindTools of qwen failed, err=%v", err) + } + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "你是一名房产经纪人，结合用户的薪酬和工作，使用 user_company、user_salary 两个 API，为其提供相关的房产信息。邮箱是必须的", + }, + { + Role: schema.User, + Content: "我的姓名是 zhangsan，我的邮箱是 zhangsan@bytedance.com，请帮我推荐一些适合我的房子。", + }, + }) + + if err != nil { + log.Fatalf("Generate of qwen failed, err=%v", err) + } + + fmt.Println(resp) + // assistant: + // tool_calls: [{0x14000275930 call_1e25169e05fc4596a55afb function {user_company {"email": "zhangsan@bytedance.com", "name": "zhangsan"}} map[]}] + // finish_reason: tool_calls + // usage: &{316 32 348} + + // ========================== + // using stream + fmt.Printf("\n\n======== Stream ========\n") + sr, err := cm.Stream(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "你是一名房产经纪人，结合用户的薪酬和工作，使用 user_company、user_salary 两个 API，为其提供相关的房产信息。邮箱是必须的", + }, + { + Role: schema.User, + Content: "我的姓名是 lisi，我的邮箱是 lisi@bytedance.com，请帮我推荐一些适合我的房子。", + }, + }) + if err != nil { + log.Fatalf("Stream of qwen failed, err=%v", err) + } + + msgs := make([]*schema.Message, 0) + for { + msg, err := sr.Recv() + if err != nil { + break + } + jsonMsg, err := json.Marshal(msg) + if err != nil { + log.Fatalf("json.Marshal failed, err=%v", err) + } + fmt.Printf("%s\n", jsonMsg) + msgs = append(msgs, msg) + } + + msg, err := schema.ConcatMessages(msgs) + if err != nil { + log.Fatalf("ConcatMessages failed, err=%v", err) + } + jsonMsg, err := json.Marshal(msg) + if err != nil { + log.Fatalf("json.Marshal failed, err=%v", err) + } + fmt.Printf("final: %s\n", jsonMsg) +} + +func of[T any](t T) *T { + return &t +} + +``` + + + +## For More Details +- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) +- [Qwen Documentation](https://help.aliyun.com/zh/model-studio/use-qwen-by-calling-api) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md index d5d4f535dd5..cd0d25f5f24 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_ark.md @@ -1,30 +1,30 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - ark weight: 0 --- -A Volcengine Ark model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个 [Eino](https://github.com/cloudwego/eino) 的 Volcengine Ark 模型实现，实现了 `ToolCallingChatModel` 接口。这使得与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -This package provides two distinct models: -- **ChatModel**: For text-based and multi-modal chat completions. -- **ImageGenerationModel**: For generating images from text prompts or image. -- **ResponseAPI**: Contains methods and other services that help with interacting with the openai API. +该软件包提供了两种不同的模型： +- **ChatModel**：用于基于文本和多模态的聊天补全。 +- **ImageGenerationModel**：用于从文本提示或图像生成图像。 +- **ResponseAPI**：包含有助于与 openai API 交互的方法和其他服务。 -## Features +## 特性 -- Implements `github.com/cloudwego/eino/components/model.Model` -- Easy integration with Eino's model system -- Configurable model parameters -- Support for both chat completion, image generation and response api -- Support for streaming responses -- Custom response parsing support -- Flexible model configuration +- 实现了 `github.com/cloudwego/eino/components/model.Model` +- 轻松与 Eino 的模型系统集成 +- 可配置的模型参数 +- 支持聊天补全、图像生成和响应 API +- 支持流式响应 +- 支持自定义响应解析 +- 灵活的模型配置 -## Installation +## 快速安装 ```bash go get github.com/cloudwego/eino-ext/components/model/ark@latest @@ -32,13 +32,13 @@ go get github.com/cloudwego/eino-ext/components/model/ark@latest --- -## Chat Completion +## 聊天 -This model is used for standard chat and text generation tasks. +该模型用于标准聊天和文本生成任务。 -### Quick Start +### 快速开始 -Here's a quick example of how to use the `ChatModel`: +以下是如何使用 `ChatModel` 的快速示例： ```go package main @@ -81,9 +81,8 @@ func main() { } log.Printf("generate output: \n") - log.Printf(" request_id: %s\n") respBody, _ := json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf(" body: %s \n", string(respBody)) sr, err := chatModel.Stream(ctx, inMsgs) if err != nil { @@ -109,15 +108,15 @@ func main() { } log.Printf("stream final output: \n") - log.Printf(" request_id: %s\n") + log.Printf(" request_id: %s \n") respBody, _ = json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf("body: %s \n", string(respBody)) } ``` -### Configuration +### 配置 -The `ChatModel` can be configured using the `ark.ChatModelConfig` struct: +`ChatModel` 可以使用 `ark.ChatModelConfig` 结构进行配置： ```go type ChatModelConfig struct { @@ -189,25 +188,15 @@ type ChatModelConfig struct { } ``` -### Request Options - -The `ChatModel` supports various request options to customize the behavior of API calls. Here are the available options: - -```go -// WithCustomHeader sets custom headers for a single request -// the headers will override all the headers given in ChatModelConfig.CustomHeader -func WithCustomHeader(m map[string]string) model.Option {} -``` - --- -## Image Generation +## 图像生成 -This model is used specifically for generating images from text prompts. +该模型专门用于从文本提示生成图像。 -### Quick Start +### 快速开始 -Here's a quick example of how to use the `ImageGenerationModel`: +以下是如何使用 `ImageGenerationModel` 的快速示例： ```go package main @@ -247,16 +236,17 @@ func main() { log.Fatalf("Generate failed, err=%v", err) } - log.Printf("\ngenerate output: \n") + log.Printf("generate output:") + log.Printf(" request_id: %s", ark.GetArkRequestID(msg)) respBody, _ := json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf(" body: %s", string(respBody)) sr, err := imageGenerationModel.Stream(ctx, inMsgs) if err != nil { log.Fatalf("Stream failed, err=%v", err) } - log.Printf("stream output: \n") + log.Printf("stream output:") index := 0 for { msgChunk, err := sr.Recv() @@ -268,68 +258,92 @@ func main() { } respBody, _ = json.MarshalIndent(msgChunk, " ", " ") - log.Printf("stream chunk %d: body: %s\n", index, string(respBody)) + log.Printf("stream chunk %d: body: %s \n", index, string(respBody)) index++ } } ``` -### Configuration +### 配置 -The `ImageGenerationModel` can be configured using the `ark.ImageGenerationConfig` struct: +`ImageGenerationModel` 可以使用 `ark.ImageGenerationConfig` 结构进行配置： ```go -type ImageGenerationConfig struct { - // --- Authentication and basic connection settings --- - // (Timeout, HTTPClient, RetryTimes, BaseURL, Region, APIKey) - // ... - // --- Image Generation Specific Parameters --- - // Ref: https://www.volcengine.com/docs/82379/1541523 - - // Model specifies the ID of the image generation endpoint on the Ark platform. - // Required. +type ImageGenerationConfig struct { + // For authentication, APIKey is required as the image generation API only supports API Key authentication. + // For authentication details, see: https://www.volcengine.com/docs/82379/1298459 + // Required + APIKey string `json:"api_key"` + + // Model specifies the ID of endpoint on ark platform + // Required Model string `json:"model"` - + + // Timeout specifies the maximum duration to wait for API responses + // If HTTPClient is set, Timeout will not be used. + // Optional. Default: 10 minutes + Timeout *time.Duration `json:"timeout"` + + // HTTPClient specifies the client to send HTTP requests. + // If HTTPClient is set, Timeout will not be used. + // Optional. Default &http.Client{Timeout: Timeout} + HTTPClient *http.Client `json:"http_client"` + + // RetryTimes specifies the number of retry attempts for failed API calls + // Optional. Default: 2 + RetryTimes *int `json:"retry_times"` + + // BaseURL specifies the base URL for Ark service + // Optional. Default: "https://ark.cn-beijing.volces.com/api/v3" + BaseURL string `json:"base_url"` + + // Region specifies the region where Ark service is located + // Optional. Default: "cn-beijing" + Region string `json:"region"` + + // The following fields correspond to Ark's image generation API parameters + // Ref: https://www.volcengine.com/docs/82379/1541523 + // Size specifies the dimensions of the generated image. - // It can be a resolution keyword (e.g., "1K", "2K", "4K") or a custom resolution - // in "{width}x{height}" format (e.g., "1920x1080"). - // When using custom resolutions, the total pixels must be between 1280x720 and 4096x4096, - // and the aspect ratio (width/height) must be between 1/16 and 16. - // Optional. Defaults to "2048x2048". - Size string `json:"size"` - - // SequentialImageGeneration determines if the model should generate a sequence of images. - // Possible values: - // - "auto": The model decides whether to generate multiple images based on the prompt. - // - "disabled": Only a single image is generated. - // Optional. Defaults to "disabled". - SequentialImageGeneration SequentialImageGeneration `json:"sequential_image_generation"` - - // SequentialImageGenerationOption sets the maximum number of images to generate when - // SequentialImageGeneration is set to "auto". - // The value must be between 1 and 15. - // Optional. Defaults to 15. - SequentialImageGenerationOption *model.SequentialImageGenerationOptions `json:"sequential_image_generation_option"` - - // ResponseFormat specifies how the generated image data is returned. - // Possible values: - // - "url": A temporary URL to download the image (valid for 24 hours). - // - "b64_json": The image data encoded as a Base64 string in the response. - // Optional. Defaults to "url". - ResponseFormat ImageResponseFormat `json:"response_format"` - - // DisableWatermark, if set to true, removes the "AI Generated" watermark - // from the bottom-right corner of the image. - // Optional. Defaults to false. - DisableWatermark bool `json:"disable_watermark"` + // It can be a resolution keyword (e.g., "1K", "2K", "4K") or a custom resolution + // in "{width}x{height}" format (e.g., "1920x1080"). + // When using custom resolutions, the total pixels must be between 1280x720 and 4096x4096, + // and the aspect ratio (width/height) must be between 1/16 and 16. + // Optional. Defaults to "2048x2048". + Size string `json:"size"` + + // SequentialImageGeneration determines if the model should generate a sequence of images. + // Possible values: + // - "auto": The model decides whether to generate multiple images based on the prompt. + // - "disabled": Only a single image is generated. + // Optional. Defaults to "disabled". + SequentialImageGeneration SequentialImageGeneration `json:"sequential_image_generation"` + + // SequentialImageGenerationOption sets the maximum number of images to generate when + // SequentialImageGeneration is set to "auto". + // The value must be between 1 and 15. + // Optional. Defaults to 15. + SequentialImageGenerationOption *model.SequentialImageGenerationOptions `json:"sequential_image_generation_option"` + + // ResponseFormat specifies how the generated image data is returned. + // Possible values: + // - "url": A temporary URL to download the image (valid for 24 hours). + // - "b64_json": The image data encoded as a Base64 string in the response. + // Optional. Defaults to "url". + ResponseFormat ImageResponseFormat `json:"response_format"` + + // DisableWatermark, if set to true, removes the "AI Generated" watermark + // from the bottom-right corner of the image. + // Optional. Defaults to false. + DisableWatermark bool `json:"disable_watermark"` } ``` -## examples +## 示例 -### generate +### 文本生成 ```go @@ -373,10 +387,10 @@ func main() { log.Fatalf("Generate failed, err=%v", err) } - log.Printf("\ngenerate output: \n") - log.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + log.Printf("generate output:") + log.Printf(" request_id: %s", ark.GetArkRequestID(msg)) respBody, _ := json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf(" body: %s", string(respBody)) sr, err := chatModel.Stream(ctx, inMsgs) if err != nil { @@ -401,15 +415,15 @@ func main() { log.Fatalf("ConcatMessages failed, err=%v", err) } - log.Printf("stream final output: \n") - log.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) + log.Printf("stream final output:") + log.Printf(" request_id: %s", ark.GetArkRequestID(msg)) respBody, _ = json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf(" body: %s \n", string(respBody)) } ``` -### generate_with_image +### 多模态支持(图片理解) ```go @@ -421,8 +435,11 @@ import ( "log" "os" - "github.com/cloudwego/eino-ext/components/model/ark" + "github.com/cloudwego/eino/components/prompt" + "github.com/cloudwego/eino/compose" "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ark" ) func main() { @@ -440,7 +457,7 @@ func main() { multiModalMsg := schema.UserMessage("") image, err := os.ReadFile("./examples/generate_with_image/eino.png") if err != nil { - log.Fatalf("os.ReadFile failed, err=%v\n", err) + log.Fatalf("os.ReadFile failed, err=%v \n", err) } imageStr := base64.StdEncoding.EncodeToString(image) @@ -469,12 +486,51 @@ func main() { log.Fatalf("Generate failed, err=%v", err) } - log.Printf("Ark ChatModel output: \n%v", resp) + log.Printf("Ark ChatModel output:%v \n", resp) + + // demonstrate how to use ChatTemplate to generate with image + imgPlaceholder := "{img}" + ctx = context.Background() + chain := compose.NewChain[map[string]any, *schema.Message]() + _ = chain.AppendChatTemplate(prompt.FromMessages(schema.FString, + &schema.Message{ + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeText, + Text: "What do you see in this image?", + }, + { + Type: schema.ChatMessagePartTypeImageURL, + Image: &schema.MessageInputImage{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: &imgPlaceholder, + MIMEType: "image/png", + }, + Detail: schema.ImageURLDetailAuto, + }, + }, + }, + })) + _ = chain.AppendChatModel(chatModel) + r, err := chain.Compile(ctx) + if err != nil { + log.Fatalf("Compile failed, err=%v", err) + } + + resp, err = r.Invoke(ctx, map[string]any{ + "img": imageStr, + }) + if err != nil { + log.Fatalf("Run failed, err=%v", err) + } + + log.Printf("Ark ChatModel output with ChatTemplate:%v \n", resp) } ``` -### stream +### 流式生成 ```go @@ -489,6 +545,7 @@ import ( "github.com/cloudwego/eino-ext/components/model/ark" "github.com/cloudwego/eino/schema" + arkModel "github.com/volcengine/volcengine-go-sdk/service/arkruntime/model" ) func main() { @@ -509,7 +566,7 @@ func main() { Role: schema.User, Content: "as a machine, how do you answer user's question?", }, - }) + }, ark.WithReasoningEffort(arkModel.ReasoningEffortHigh)) if err != nil { log.Printf("Generate failed, err=%v", err) @@ -528,7 +585,7 @@ func main() { } msgs = append(msgs, msg) if err != nil { - log.Printf("\nstream.Recv failed, err=%v", err) + log.Printf("stream.Recv failed, err=%v", err) return } fmt.Print(msg.Content) @@ -540,12 +597,98 @@ func main() { return } - log.Printf("output: %s\n", msg.Content) + log.Printf("output: %s \n", msg.Content) } ``` -### image_generate +### 工具调用 + +```go + +package main + +import ( + "context" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/ark" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + + // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 + chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ + APIKey: os.Getenv("ARK_API_KEY"), + Model: os.Getenv("ARK_MODEL_ID"), + }) + if err != nil { + log.Printf("NewChatModel failed, err=%v", err) + return + } + + err = chatModel.BindTools([]*schema.ToolInfo{ + { + Name: "user_company", + Desc: "Query the user's company and position information based on their name and email", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "The user's name", + }, + "email": { + Type: "string", + Desc: "The user's email", + }, + }), + }, + { + Name: "user_salary", + Desc: "Query the user's salary information based on their name and email", + ParamsOneOf: schema.NewParamsOneOfByParams( + map[string]*schema.ParameterInfo{ + "name": { + Type: "string", + Desc: "The user's name", + }, + "email": { + Type: "string", + Desc: "The user's email", + }, + }), + }, + }) + if err != nil { + log.Printf("BindForcedTools failed, err=%v", err) + return + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.System, + Content: "You are a real estate agent. Use the user_company and user_salary APIs to provide relevant property information based on the user's salary and job. Email is required", + }, + { + Role: schema.User, + Content: "My name is zhangsan, and my email is zhangsan@bytedance.com. Please recommend some suitable houses for me.", + }, + }) + + if err != nil { + log.Printf("Generate failed, err=%v", err) + return + } + + log.Printf("output:%v \n", resp) +} + +``` + +### 图像生成 ```go @@ -612,17 +755,18 @@ func main() { log.Fatalf("Generate failed, err=%v", err) } - log.Printf("\ngenerate output: \n") + log.Printf("generate output:") respBody, _ := json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf(" body: %s \n", string(respBody)) sr, err := imageGenerationModel.Stream(ctx, inMsgs) if err != nil { log.Fatalf("Stream failed, err=%v", err) } - log.Printf("stream output: \n") + log.Printf("stream output:") index := 0 + chunks := make([]*schema.Message, 0, 1024) for { msgChunk, err := sr.Recv() if errors.Is(err, io.EOF) { @@ -632,101 +776,26 @@ func main() { log.Fatalf("Stream Recv failed, err=%v", err) } + chunks = append(chunks, msgChunk) + respBody, _ = json.MarshalIndent(msgChunk, " ", " ") log.Printf("stream chunk %d: body: %s\n", index, string(respBody)) index++ } -} - -``` - -### intent_tool - -```go - -package main - -import ( - "context" - "log" - "os" - - "github.com/cloudwego/eino-ext/components/model/ark" - "github.com/cloudwego/eino/schema" -) - -func main() { - ctx := context.Background() - - // Get ARK_API_KEY and ARK_MODEL_ID: https://www.volcengine.com/docs/82379/1399008 - chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ - APIKey: os.Getenv("ARK_API_KEY"), - Model: os.Getenv("ARK_MODEL_ID"), - }) - if err != nil { - log.Printf("NewChatModel failed, err=%v", err) - return - } - - err = chatModel.BindTools([]*schema.ToolInfo{ - { - Name: "user_company", - Desc: "Query the user's company and position information based on their name and email", - ParamsOneOf: schema.NewParamsOneOfByParams( - map[string]*schema.ParameterInfo{ - "name": { - Type: "string", - Desc: "The user's name", - }, - "email": { - Type: "string", - Desc: "The user's email", - }, - }), - }, - { - Name: "user_salary", - Desc: "Query the user's salary information based on their name and email", - ParamsOneOf: schema.NewParamsOneOfByParams( - map[string]*schema.ParameterInfo{ - "name": { - Type: "string", - Desc: "The user's name", - }, - "email": { - Type: "string", - Desc: "The user's email", - }, - }), - }, - }) - if err != nil { - log.Printf("BindForcedTools failed, err=%v", err) - return - } - - resp, err := chatModel.Generate(ctx, []*schema.Message{ - { - Role: schema.System, - Content: "You are a real estate agent. Use the user_company and user_salary APIs to provide relevant property information based on the user's salary and job. Email is required", - }, - { - Role: schema.User, - Content: "My name is zhangsan, and my email is zhangsan@bytedance.com. Please recommend some suitable houses for me.", - }, - }) + msg, err = schema.ConcatMessages(chunks) if err != nil { - log.Printf("Generate failed, err=%v", err) - return + log.Fatalf("ConcatMessages failed, err=%v", err) } - - log.Printf("output: \n%v", resp) + log.Printf("stream final output:") + log.Printf(" request_id: %s \n", ark.GetArkRequestID(msg)) + respBody, _ = json.MarshalIndent(msg, " ", " ") + log.Printf(" body: %s \n", string(respBody)) } ``` -### prefixcache-contextapi +### ContextAPI 前缀缓存 ```go @@ -815,7 +884,7 @@ func main() { ``` -### prefixcache-responsesapi +### Response API 前缀缓存 ```go @@ -827,8 +896,6 @@ import ( "log" "os" - arkModel "github.com/volcengine/volcengine-go-sdk/service/arkruntime/model" - "github.com/cloudwego/eino/schema" "github.com/cloudwego/eino-ext/components/model/ark" @@ -841,44 +908,111 @@ func main() { chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ APIKey: os.Getenv("ARK_API_KEY"), Model: os.Getenv("ARK_MODEL_ID"), + Cache: &ark.CacheConfig{ + APIType: ptrOf(ark.ResponsesAPI), + }, }) if err != nil { log.Fatalf("NewChatModel failed, err=%v", err) } - chatModelWithTools, err := chatModel.WithTools([]*schema.ToolInfo{ + err = chatModel.BindTools([]*schema.ToolInfo{ { - Name: "get_weather", - Desc: "Get the current weather in a given location", + Name: "article_content_extractor", + Desc: "Extract key statements and chapter summaries from the provided article content", ParamsOneOf: schema.NewParamsOneOfByParams( map[string]*schema.ParameterInfo{ - "location": { - Type: "string", - Desc: "The city and state, e.g. San Francisco, CA", + "content": { + Type: schema.String, + Desc: "The full article content to analyze and extract key information from", + Required: true, }, - }, - ), + }), }, }) + if err != nil { - log.Fatalf("WithTools failed, err=%v", err) + log.Fatalf("BindTools failed, err=%v", err) } - thinking := &arkModel.Thinking{ - Type: arkModel.ThinkingTypeDisabled, + // create response prefix cache, note: more than 1024 tokens are required, otherwise the prefix cache cannot be created + cacheInfo, err := chatModel.CreatePrefixCache(ctx, []*schema.Message{ + schema.SystemMessage(`Once upon a time, in a quaint little village surrounded by vast green forests and blooming meadows, there lived a spirited young girl known as Little Red Riding Hood. She earned her name from the vibrant red cape that her beloved grandmother had sewn for her, a gift that she cherished deeply. This cape was more than just a piece of clothing; it was a symbol of the bond between her and her grandmother, who lived on the other side of the great woods, near a sparkling brook that bubbled merrily all year round. + + One sunny morning, Little Red Riding Hood's mother called her into the cozy kitchen, where the aroma of freshly baked bread filled the air. “My dear,” she said, “your grandmother isn’t feeling well today. I want you to take her this basket of treats. There are some delicious cakes, a jar of honey, and her favorite herbal tea. Can you do that for me?” + + Little Red Riding Hood’s eyes sparkled with excitement as she nodded eagerly. “Yes, Mama! I’ll take good care of them!” Her mother handed her a beautifully woven basket, filled to the brim with goodies, and reminded her, “Remember to stay on the path and don’t talk to strangers.” + + “I promise, Mama!” she replied confidently, pulling her red hood over her head and setting off on her adventure. The sun shone brightly, and birds chirped merrily as she walked, making her feel like she was in a fairy tale. + + As she journeyed through the woods, the tall trees whispered secrets to one another, and colorful flowers danced in the gentle breeze. Little Red Riding Hood was so enchanted by the beauty around her that she began to hum a tune, her voice harmonizing with the sounds of nature. + + However, unbeknownst to her, lurking in the shadows was a cunning wolf. The wolf was known throughout the forest for his deceptive wit and insatiable hunger. He watched Little Red Riding Hood with keen interest, contemplating his next meal. + + “Good day, little girl!” the wolf called out, stepping onto the path with a friendly yet sly smile. + + Startled, she halted and took a step back. “Hello there! I’m just on my way to visit my grandmother,” she replied, clutching the basket tightly. + + “Ah, your grandmother! I know her well,” the wolf said, his eyes glinting with mischief. “Why don’t you pick some lovely flowers for her? I’m sure she would love them, and I’m sure there are many beautiful ones just off the path.” + + Little Red Riding Hood hesitated for a moment but was easily convinced by the wolf’s charming suggestion. “That’s a wonderful idea! Thank you!” she exclaimed, letting her curiosity pull her away from the safety of the path. As she wandered deeper into the woods, her gaze fixed on the vibrant blooms, the wolf took a shortcut towards her grandmother’s house. + + When the wolf arrived at Grandma’s quaint cottage, he knocked on the door with a confident swagger. “It’s me, Little Red Riding Hood!” he shouted in a high-pitched voice to mimic the girl. + + “Come in, dear!” came the frail voice of the grandmother, who had been resting on her cozy bed, wrapped in warm blankets. The wolf burst through the door, his eyes gleaming with the thrill of his plan. + + With astonishing speed, the wolf gulped down the unsuspecting grandmother whole. Afterward, he dressed in her nightgown, donning her nightcap and climbing into her bed. He lay there, waiting for Little Red Riding Hood to arrive, concealing his wicked smile behind a facade of innocence. + + Meanwhile, Little Red Riding Hood was merrily picking flowers, completely unaware of the impending danger. After gathering a beautiful bouquet of wildflowers, she finally made her way back to the path and excitedly skipped towards her grandmother’s cottage. + + Upon arriving, she noticed the door was slightly ajar. “Grandmother, it’s me!” she called out, entering the dimly lit home. It was silent, with only the faint sound of an old clock ticking in the background. She stepped into the small living room, a feeling of unease creeping over her. + + “Grandmother, are you here?” she asked, peeking into the bedroom. There, she saw a figure lying under the covers. + + “Grandmother, what big ears you have!” she exclaimed, taking a few cautious steps closer. + + “All the better to hear you with, my dear,” the wolf replied in a voice that was deceptively sweet. + + “Grandmother, what big eyes you have!” Little Red Riding Hood continued, now feeling an unsettling chill in the air. + + “All the better to see you with, my dear,” the wolf said, his eyes narrowing as he tried to contain his glee. + + “Grandmother, what big teeth you have!” she exclaimed, the terror flooding her senses as she began to realize this was no ordinary visit. + + “All the better to eat you with!” the wolf roared, springing out of the bed with startling speed. + + Just as the wolf lunged towards her, a brave woodsman, who had been passing by the cottage and heard the commotion, burst through the door. His strong presence was a beacon of hope in the dire situation. “Stay back, wolf!” he shouted with authority, brandishing his axe. + + The wolf, taken aback by the sudden intrusion, hesitated for a moment. Before he could react, the woodsman swung his axe with determination, and with a swift motion, he drove the wolf away, rescuing Little Red Riding Hood and her grandmother from certain doom. + + Little Red Riding Hood was shaking with fright, but relief washed over her as the woodsman helped her grandmother out from behind the bed where the wolf had hidden her. The grandmother, though shaken, was immensely grateful to the woodsman for his bravery. “Thank you so much! You saved us!” she cried, embracing him warmly. + + Little Red Riding Hood, still in shock but filled with gratitude, looked up at the woodsman and said, “I promise I will never stray from the path again. Thank you for being our hero!” + + From that day on, the woodland creatures spoke of the brave woodsman who saved Little Red Riding Hood and her grandmother. Little Red Riding Hood learned a valuable lesson about being cautious and listening to her mother’s advice. The bond between her and her grandmother grew stronger, and they often reminisced about that day’s adventure over cups of tea, surrounded by cookies and laughter. + + To ensure safety, Little Red Riding Hood always took extra precautions when traveling through the woods, carrying a small whistle her grandmother had given her. It would alert anyone nearby if she ever found herself in trouble again. + + And so, in the heart of that small village, life continued, filled with love, laughter, and the occasional adventure, as Little Red Riding Hood and her grandmother thrived, forever grateful for the friendship of the woodsman who had acted as their guardian that fateful day. + + And they all lived happily ever after. + + The end.`), + }, 300) + if err != nil { + log.Fatalf("CreatePrefixCache failed, err=%v", err) } + + // use cache information in subsequent requests cacheOpt := &ark.CacheOption{ - APIType: ark.ResponsesAPI, - SessionCache: &ark.SessionCacheConfig{ - EnableCache: true, - TTL: 86400, - }, + APIType: ark.ResponsesAPI, + HeadPreviousResponseID: &cacheInfo.ResponseID, } - outMsg, err := chatModelWithTools.Generate(ctx, []*schema.Message{ - schema.UserMessage("my name is megumin"), - }, ark.WithThinking(thinking), - ark.WithCache(cacheOpt)) + outMsg, err := chatModel.Generate(ctx, []*schema.Message{ + schema.UserMessage("What is the main idea expressed above？"), + }, ark.WithCache(cacheOpt)) + if err != nil { log.Fatalf("Generate failed, err=%v", err) } @@ -888,27 +1022,19 @@ func main() { log.Fatalf("not found response id in message") } - msg, err := chatModelWithTools.Generate(ctx, []*schema.Message{ - schema.UserMessage("what is my name?"), - }, ark.WithThinking(thinking), - ark.WithCache(&ark.CacheOption{ - APIType: ark.ResponsesAPI, - HeadPreviousResponseID: &respID, - }), - ) - if err != nil { - log.Fatalf("Generate failed, err=%v", err) - } - log.Printf("\ngenerate output: \n") - log.Printf(" request_id: %s\n", ark.GetArkRequestID(msg)) - respBody, _ := json.MarshalIndent(msg, " ", " ") + log.Printf(" request_id: %s\n", respID) + respBody, _ := json.MarshalIndent(outMsg, " ", " ") log.Printf(" body: %s\n", string(respBody)) } +func ptrOf[T any](v T) *T { + return &v + +} ``` -### sessioncache-contextapi +### ContextAPI Session 缓存 ```go @@ -1028,8 +1154,8 @@ func main() { ``` -### sessioncache-responsesapi +### ResponseAPI Session缓存 ```go package main @@ -1043,8 +1169,9 @@ import ( arkModel "github.com/volcengine/volcengine-go-sdk/service/arkruntime/model" - "github.com/cloudwego/eino-ext/components/model/ark" "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/ark" ) func main() { @@ -1054,6 +1181,12 @@ func main() { chatModel, err := ark.NewChatModel(ctx, &ark.ChatModelConfig{ APIKey: os.Getenv("ARK_API_KEY"), Model: os.Getenv("ARK_MODEL_ID"), + Cache: &ark.CacheConfig{ + SessionCache: &ark.SessionCacheConfig{ + EnableCache: true, + TTL: 86400, + }, + }, }) if err != nil { log.Fatalf("NewChatModel failed, err=%v", err) @@ -1114,7 +1247,7 @@ func main() { -## For More Details +## 更多信息 - [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) - [Volcengine Ark Model Documentation](https://www.volcengine.com/docs/82379/1263272) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_arkbot.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_arkbot.md index 56efedd6c5a..e6f2f9175c8 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_arkbot.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_arkbot.md @@ -1,36 +1,35 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - arkbot weight: 0 --- -A Volcengine Ark Bot implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个为 [Eino](https://github.com/cloudwego/eino) 实现的 Volcengine Ark Bot，它实现了 `ToolCallingChatModel` 接口。这使得能够与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -## Features +## 特性 -- Implements `github.com/cloudwego/eino/components/model.ToolCallingChatModel` -- Easy integration with Eino's model system -- Configurable model parameters -- Support for chat completion -- Support for streaming responses -- Custom response parsing support -- Flexible model configuration +- 实现了 `github.com/cloudwego/eino/components/model.ToolCallingChatModel` +- 轻松与 Eino 的模型系统集成 +- 可配置的模型参数 +- 支持聊天补全 +- 支持流式响应 +- 支持自定义响应解析 +- 灵活的模型配置 -## Installation +## 安装 ```bash go get github.com/cloudwego/eino-ext/components/model/arkbot@latest ``` -## Quick Start +## 快速开始 -Here's a quick example of how to use the Ark Bot: +以下是如何使用 Ark Bot 的快速示例： ```go - package main import ( @@ -69,30 +68,28 @@ func main() { log.Fatalf("Generate failed, err=%v", err) } - log.Printf("generate output: \n") - log.Printf(" request_id: %s\n", arkbot.GetArkRequestID(msg)) + log.Printf("generate output:") + log.Printf(" request_id: %s", arkbot.GetArkRequestID(msg)) if bu, ok := arkbot.GetBotUsage(msg); ok { bbu, _ := json.Marshal(bu) - log.Printf(" bot_usage: %s\n", string(bbu)) + log.Printf(" bot_usage: %s \n", string(bbu)) } if ref, ok := arkbot.GetBotChatResultReference(msg); ok { bRef, _ := json.Marshal(ref) log.Printf(" bot_chat_result_reference: %s\n", bRef) } respBody, _ := json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf(" body: %s \n", string(respBody)) } - - ``` -## Configuration +## 配置 -The model can be configured using the `arkbot.Config` struct: +可以使用 `arkbot.Config` 结构体配置模型： ```go - - type Config struct { + +type Config struct { // Timeout specifies the maximum duration to wait for API responses // If HTTPClient is set, Timeout will not be used. // Optional. Default: 10 minutes @@ -176,9 +173,9 @@ The model can be configured using the `arkbot.Config` struct: } ``` -## Request Options +## 请求选项 -The Ark model supports various request options to customize the behavior of API calls. Here are the available options: +Ark 模型支持各种请求选项以自定义 API 调用的行为。以下是可用的选项： ```go // WithCustomHeader sets custom headers for a single request @@ -186,13 +183,11 @@ The Ark model supports various request options to customize the behavior of API func WithCustomHeader(m map[string]string) model.Option {} ``` +## 示例 -## examples - -### generate +### 文本生成 ```go - package main import ( @@ -231,26 +226,24 @@ func main() { log.Fatalf("Generate failed, err=%v", err) } - log.Printf("generate output: \n") - log.Printf(" request_id: %s\n", arkbot.GetArkRequestID(msg)) + log.Printf("generate output:\n") + log.Printf(" request_id: %s \n", arkbot.GetArkRequestID(msg)) if bu, ok := arkbot.GetBotUsage(msg); ok { bbu, _ := json.Marshal(bu) - log.Printf(" bot_usage: %s\n", string(bbu)) + log.Printf(" bot_usage: %s \n", string(bbu)) } if ref, ok := arkbot.GetBotChatResultReference(msg); ok { bRef, _ := json.Marshal(ref) - log.Printf(" bot_chat_result_reference: %s\n", bRef) + log.Printf(" bot_chat_result_reference: %s \n", bRef) } respBody, _ := json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf(" body: %s \n", string(respBody)) } - ``` -### stream +### 流式生成 ```go - package main import ( @@ -303,7 +296,7 @@ func main() { } msgs = append(msgs, msg) if err != nil { - log.Printf("\nstream.Recv failed, err=%v", err) + log.Printf("stream.Recv failed, err=%v", err) return } fmt.Print(msg.Content) @@ -316,24 +309,21 @@ func main() { } log.Printf("generate output: \n") - log.Printf(" request_id: %s\n", arkbot.GetArkRequestID(msg)) + log.Printf(" request_id: %s \n", arkbot.GetArkRequestID(msg)) if bu, ok := arkbot.GetBotUsage(msg); ok { bbu, _ := json.Marshal(bu) - log.Printf(" bot_usage: %s\n", string(bbu)) + log.Printf(" bot_usage: %s \n", string(bbu)) } if ref, ok := arkbot.GetBotChatResultReference(msg); ok { bRef, _ := json.Marshal(ref) - log.Printf(" bot_chat_result_reference: %s\n", bRef) + log.Printf(" bot_chat_result_reference: %s \n", bRef) } respBody, _ := json.MarshalIndent(msg, " ", " ") - log.Printf(" body: %s\n", string(respBody)) + log.Printf(" body: %s \n", string(respBody)) } - ``` - - -## For More Details +## 更多信息 - [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) - [Volcengine Ark Model Documentation](https://www.volcengine.com/docs/82379/1263272) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_claude.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_claude.md index dd44bc9220b..5083bfe6cc9 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_claude.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_claude.md @@ -1,33 +1,33 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - claude weight: 0 --- -A Claude model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个为 [Eino](https://github.com/cloudwego/eino) 实现的 Claude 模型，它实现了 `ToolCallingChatModel` 接口。这使得能够与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -## Features +## 特性 -- Implements `github.com/cloudwego/eino/components/model.Model` -- Easy integration with Eino's model system -- Configurable model parameters -- Support for chat completion -- Support for streaming responses -- Custom response parsing support -- Flexible model configuration +- 实现了 `github.com/cloudwego/eino/components/model.Model` +- 轻松与 Eino 的模型系统集成 +- 可配置的模型参数 +- 支持聊天补全 +- 支持流式响应 +- 支持自定义响应解析 +- 灵活的模型配置 -## Installation +## 安装 ```bash go get github.com/cloudwego/eino-ext/components/model/claude@latest ``` -## Quick Start +## 快速开始 -Here's a quick example of how to use the Claude model: +以下是如何使用 Claude 模型的快速示例： ```go package main @@ -104,12 +104,11 @@ func main() { resp.ResponseMeta.Usage.TotalTokens) } } - ``` -## Configuration +## 配置 -The model can be configured using the `claude.ChatModelConfig` struct: +可以使用 `claude.ChatModelConfig` 结构体配置模型： ```go type Config struct { @@ -190,16 +189,11 @@ type Config struct { } ``` +## 示例 - - - -## examples - -### generate +### 文本生成 ```go - package main import ( @@ -274,13 +268,11 @@ func main() { resp.ResponseMeta.Usage.TotalTokens) } } - ``` -### generate_with_image +### 多模态支持(图片理解) ```go - package main import ( @@ -360,13 +352,11 @@ func main() { func of[T any](v T) *T { return &v } - ``` -### stream +### 流式生成 ```go - package main import ( @@ -461,13 +451,136 @@ func main() { } fmt.Println("\n----------") } - ``` -### claude_prompt_cache +### 工具调用 ```go +package main + +import ( + "context" + "fmt" + + "io" + "log" + "os" + + "github.com/cloudwego/eino-ext/components/model/claude" + "github.com/cloudwego/eino/schema" + "github.com/eino-contrib/jsonschema" + orderedmap "github.com/wk8/go-ordered-map/v2" +) + +func main() { + ctx := context.Background() + apiKey := os.Getenv("CLAUDE_API_KEY") + modelName := os.Getenv("CLAUDE_MODEL") + baseURL := os.Getenv("CLAUDE_BASE_URL") + if apiKey == "" { + log.Fatal("CLAUDE_API_KEY environment variable is not set") + } + + var baseURLPtr *string = nil + if len(baseURL) > 0 { + baseURLPtr = &baseURL + } + + // Create a Claude model + cm, err := claude.NewChatModel(ctx, &claude.Config{ + // if you want to use Aws Bedrock Service, set these four field. + // ByBedrock: true, + // AccessKey: "", + // SecretAccessKey: "", + // Region: "us-west-2", + APIKey: apiKey, + // Model: "claude-3-5-sonnet-20240620", + BaseURL: baseURLPtr, + Model: modelName, + MaxTokens: 3000, + }) + if err != nil { + log.Fatalf("NewChatModel of claude failed, err=%v", err) + } + + _, err = cm.WithTools([]*schema.ToolInfo{ + { + Name: "get_weather", + Desc: "Get current weather information for a city", + ParamsOneOf: schema.NewParamsOneOfByJSONSchema(&jsonschema.Schema{ + Type: "object", + Properties: orderedmap.New[string, *jsonschema.Schema](orderedmap.WithInitialData[string, *jsonschema.Schema]( + orderedmap.Pair[string, *jsonschema.Schema]{ + Key: "city", + Value: &jsonschema.Schema{ + Type: "string", + Description: "The city name", + }, + }, + orderedmap.Pair[string, *jsonschema.Schema]{ + Key: "unit", + Value: &jsonschema.Schema{ + Type: "string", + Enum: []interface{}{"celsius", "fahrenheit"}, + }, + }, + )), + Required: []string{"city"}, + }), + }, + }) + if err != nil { + log.Printf("Bind tools error: %v", err) + return + } + + streamResp, err := cm.Stream(ctx, []*schema.Message{ + schema.SystemMessage("You are a helpful AI assistant. Be concise in your responses."), + schema.UserMessage("call 'get_weather' to query what's the weather like in Paris today? Please use Celsius."), + }) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + + msgs := make([]*schema.Message, 0) + for { + msg, err := streamResp.Recv() + if err == io.EOF { + break + } + if err != nil { + log.Fatalf("Stream receive error: %v", err) + } + msgs = append(msgs, msg) + } + resp, err := schema.ConcatMessages(msgs) + if err != nil { + log.Fatalf("Concat error: %v", err) + } + + fmt.Printf("assistant content:\n %v\n----------\n", resp.Content) + if len(resp.ToolCalls) > 0 { + fmt.Printf("Function called: %s\n", resp.ToolCalls[0].Function.Name) + fmt.Printf("Arguments: %s\n", resp.ToolCalls[0].Function.Arguments) + + weatherResp, err := cm.Generate(ctx, []*schema.Message{ + schema.UserMessage("What's the weather like in Paris today? Please use Celsius."), + resp, + schema.ToolMessage(`{"temperature": 18, "condition": "sunny"}`, resp.ToolCalls[0].ID), + }) + if err != nil { + log.Printf("Generate error: %v", err) + return + } + fmt.Printf("Final response: %s\n", weatherResp.Content) + } +} +``` + +### Claude 提示词缓存 +```go package main import ( @@ -739,136 +852,9 @@ func sessionCache() { ``` -### function_call - -```go - -package main - -import ( - "context" - "fmt" - - "io" - "log" - "os" - - "github.com/cloudwego/eino-ext/components/model/claude" - "github.com/cloudwego/eino/schema" - "github.com/eino-contrib/jsonschema" - orderedmap "github.com/wk8/go-ordered-map/v2" -) - -func main() { - ctx := context.Background() - apiKey := os.Getenv("CLAUDE_API_KEY") - modelName := os.Getenv("CLAUDE_MODEL") - baseURL := os.Getenv("CLAUDE_BASE_URL") - if apiKey == "" { - log.Fatal("CLAUDE_API_KEY environment variable is not set") - } - - var baseURLPtr *string = nil - if len(baseURL) > 0 { - baseURLPtr = &baseURL - } - - // Create a Claude model - cm, err := claude.NewChatModel(ctx, &claude.Config{ - // if you want to use Aws Bedrock Service, set these four field. - // ByBedrock: true, - // AccessKey: "", - // SecretAccessKey: "", - // Region: "us-west-2", - APIKey: apiKey, - // Model: "claude-3-5-sonnet-20240620", - BaseURL: baseURLPtr, - Model: modelName, - MaxTokens: 3000, - }) - if err != nil { - log.Fatalf("NewChatModel of claude failed, err=%v", err) - } - - _, err = cm.WithTools([]*schema.ToolInfo{ - { - Name: "get_weather", - Desc: "Get current weather information for a city", - ParamsOneOf: schema.NewParamsOneOfByJSONSchema(&jsonschema.Schema{ - Type: "object", - Properties: orderedmap.New[string, *jsonschema.Schema](orderedmap.WithInitialData[string, *jsonschema.Schema]( - orderedmap.Pair[string, *jsonschema.Schema]{ - Key: "city", - Value: &jsonschema.Schema{ - Type: "string", - Description: "The city name", - }, - }, - orderedmap.Pair[string, *jsonschema.Schema]{ - Key: "unit", - Value: &jsonschema.Schema{ - Type: "string", - Enum: []interface{}{"celsius", "fahrenheit"}, - }, - }, - )), - Required: []string{"city"}, - }), - }, - }) - if err != nil { - log.Printf("Bind tools error: %v", err) - return - } - - streamResp, err := cm.Stream(ctx, []*schema.Message{ - schema.SystemMessage("You are a helpful AI assistant. Be concise in your responses."), - schema.UserMessage("call 'get_weather' to query what's the weather like in Paris today? Please use Celsius."), - }) - if err != nil { - log.Printf("Generate error: %v", err) - return - } - - msgs := make([]*schema.Message, 0) - for { - msg, err := streamResp.Recv() - if err == io.EOF { - break - } - if err != nil { - log.Fatalf("Stream receive error: %v", err) - } - msgs = append(msgs, msg) - } - resp, err := schema.ConcatMessages(msgs) - if err != nil { - log.Fatalf("Concat error: %v", err) - } - - fmt.Printf("assistant content:\n %v\n----------\n", resp.Content) - if len(resp.ToolCalls) > 0 { - fmt.Printf("Function called: %s\n", resp.ToolCalls[0].Function.Name) - fmt.Printf("Arguments: %s\n", resp.ToolCalls[0].Function.Arguments) - - weatherResp, err := cm.Generate(ctx, []*schema.Message{ - schema.UserMessage("What's the weather like in Paris today? Please use Celsius."), - resp, - schema.ToolMessage(`{"temperature": 18, "condition": "sunny"}`, resp.ToolCalls[0].ID), - }) - if err != nil { - log.Printf("Generate error: %v", err) - return - } - fmt.Printf("Final response: %s\n", weatherResp.Content) - } -} - -``` - -## For More Details +## 更多信息 - [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) - [Claude Documentation](https://docs.claude.com/en/api/messages) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_deepseek.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_deepseek.md index 8d04ee1680c..64d5011e55d 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_deepseek.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_deepseek.md @@ -1,33 +1,33 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - deepseek weight: 0 --- -A DeepSeek model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个针对 [Eino](https://github.com/cloudwego/eino) 的 DeepSeek 模型实现，实现了 `ToolCallingChatModel` 接口。这使得能够与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -## Features +## 特性 -- Implements `github.com/cloudwego/eino/components/model.Model` -- Easy integration with Eino's model system -- Configurable model parameters -- Support for chat completion -- Support for streaming responses -- Custom response parsing support -- Flexible model configuration +- 实现了 `github.com/cloudwego/eino/components/model.Model` +- 轻松与 Eino 的模型系统集成 +- 可配置的模型参数 +- 支持聊天补全 +- 支持流式响应 +- 支持自定义响应解析 +- 灵活的模型配置 -## Installation +## 安装 ```bash go get github.com/cloudwego/eino-ext/components/model/deepseek@latest ``` -## Quick Start +## 快速开始 -Here's a quick example of how to use the DeepSeek model: +以下是如何使用 DeepSeek 模型的快速示例： ```go package main @@ -81,106 +81,100 @@ func main() { if !ok { fmt.Printf("Unexpected: non-reasoning") } else { - fmt.Printf("Reasoning Content: %s\n", reasoning) + fmt.Printf("Reasoning Content: %s \n", reasoning) } fmt.Printf("Assistant: %s\n", resp.Content) if resp.ResponseMeta != nil && resp.ResponseMeta.Usage != nil { - fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total)\n", + fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total) \n", resp.ResponseMeta.Usage.PromptTokens, resp.ResponseMeta.Usage.CompletionTokens, - resp.ResponseMeta.Usage.TotalTokens) + resp.ResponseMetaUsage.TotalTokens) } } ``` -## Configuration +## 配置 -The model can be configured using the `deepseek.ChatModelConfig` struct: +可以使用 `deepseek.ChatModelConfig` 结构体配置模型： ```go - type ChatModelConfig struct { - // APIKey is your authentication key - // Required - APIKey string `json:"api_key"` - - // Timeout specifies the maximum duration to wait for API responses - // Optional. Default: 5 minutes - Timeout time.Duration `json:"timeout"` - - // HTTPClient specifies the client to send HTTP requests. - // Optional. Default http.DefaultClient - HTTPClient *http.Client `json:"http_client"` - - // BaseURL is your custom deepseek endpoint url - // Optional. Default: https://api.deepseek.com/ - BaseURL string `json:"base_url"` - - // Path sets the path for the API request. Defaults to "chat/completions", if not set. - // Example usages would be "/c/chat/" or any http after the baseURL extension - // Path 用于设置 API 请求的路径。如果未设置，则默认为 "chat/completions"。 - // 用法示例可以是 "/c/chat/" 或 baseURL 之后的任何 http 路径。 - Path string `json:"path"` - - // The following fields correspond to DeepSeek's chat API parameters - // Ref: https://api-docs.deepseek.com/api/create-chat-completion - - // Model specifies the ID of the model to use - // Required - Model string `json:"model"` - - // MaxTokens limits the maximum number of tokens that can be generated in the chat completion - // Range: [1, 8192]. - // Optional. Default: 4096 - MaxTokens int `json:"max_tokens,omitempty"` - - // Temperature specifies what sampling temperature to use - // Generally recommend altering this or TopP but not both. - // Range: [0.0, 2.0]. Higher values make output more random - // Optional. Default: 1.0 - Temperature float32 `json:"temperature,omitempty"` - - // TopP controls diversity via nucleus sampling - // Generally recommend altering this or Temperature but not both. - // Range: [0.0, 1.0]. Lower values make output more focused - // Optional. Default: 1.0 - TopP float32 `json:"top_p,omitempty"` - - // Stop sequences where the API will stop generating further tokens - // Optional. Example: []string{"\n", "User:"} - Stop []string `json:"stop,omitempty"` - - // PresencePenalty prevents repetition by penalizing tokens based on presence - // Range: [-2.0, 2.0]. Positive values increase likelihood of new topics - // Optional. Default: 0 - PresencePenalty float32 `json:"presence_penalty,omitempty"` - - // ResponseFormat specifies the format of the model's response - // Optional. Use for structured outputs - ResponseFormatType ResponseFormatType `json:"response_format_type,omitempty"` - - // FrequencyPenalty prevents repetition by penalizing tokens based on frequency - // Range: [-2.0, 2.0]. Positive values decrease likelihood of repetition - // Optional. Default: 0 - FrequencyPenalty float32 `json:"frequency_penalty,omitempty"` - - // LogProbs specifies whether to return log probabilities of the output tokens. - LogProbs bool `json:"log_probs"` - - // TopLogProbs specifies the number of most likely tokens to return at each token position, each with an associated log probability. - TopLogProbs int `json:"top_log_probs"` + // APIKey is your authentication key + // Required + APIKey string `json:"api_key"` + + // Timeout specifies the maximum duration to wait for API responses + // Optional. Default: 5 minutes + Timeout time.Duration `json:"timeout"` + + // HTTPClient specifies the client to send HTTP requests. + // Optional. Default http.DefaultClient + HTTPClient *http.Client `json:"http_client"` + + // BaseURL is your custom deepseek endpoint url + // Optional. Default: https://api.deepseek.com/ + BaseURL string `json:"base_url"` + + // Path sets the path for the API request. Defaults to "chat/completions", if not set. + // Example usages would be "/c/chat/" or any http after the baseURL extension + // Path 用于设置 API 请求的路径。如果未设置，则默认为 "chat/completions"。 + // 用法示例可以是 "/c/chat/" 或 baseURL 之后的任何 http 路径。 + Path string `json:"path"` + + // The following fields correspond to DeepSeek's chat API parameters + // Ref: https://api-docs.deepseek.com/api/create-chat-completion + + // Model specifies the ID of the model to use + // Required + Model string `json:"model"` + + // MaxTokens limits the maximum number of tokens that can be generated in the chat completion + // Range: [1, 8192]. + // Optional. Default: 4096 + MaxTokens int `json:"max_tokens,omitempty"` + + // Temperature specifies what sampling temperature to use + // Generally recommend altering this or TopP but not both. + // Range: [0.0, 2.0]. Higher values make output more random + // Optional. Default: 1.0 + Temperature float32 `json:"temperature,omitempty"` + + // TopP controls diversity via nucleus sampling + // Generally recommend altering this or Temperature but not both. + // Range: [0.0, 1.0]. Lower values make output more focused + // Optional. Default: 1.0 + TopP float32 `json:"top_p,omitempty"` + + // Stop sequences where the API will stop generating further tokens + // Optional. Example: []string{"\n", "User:"} + Stop []string `json:"stop,omitempty"` + + // PresencePenalty prevents repetition by penalizing tokens based on presence + // Range: [-2.0, 2.0]. Positive values increase likelihood of new topics + // Optional. Default: 0 + PresencePenalty float32 `json:"presence_penalty,omitempty"` + + // ResponseFormat specifies the format of the model's response + // Optional. Use for structured outputs + ResponseFormatType ResponseFormatType `json:"response_format_type,omitempty"` + + // FrequencyPenalty prevents repetition by penalizing tokens based on frequency + // Range: [-2.0, 2.0]. Positive values decrease likelihood of repetition + // Optional. Default: 0 + FrequencyPenalty float32 `json:"frequency_penalty,omitempty"` + + // LogProbs specifies whether to return log probabilities of the output tokens. + LogProbs bool `json:"log_probs"` + + // TopLogProbs specifies the number of most likely tokens to return at each token position, each with an associated log probability. + TopLogProbs int `json:"top_log_probs"` } - ``` +## 示例 - - - -## examples - -### generate +### 文本生成 ```go @@ -237,9 +231,9 @@ func main() { } else { fmt.Printf("Reasoning Content: %s\n", reasoning) } - fmt.Printf("Assistant: %s\n", resp.Content) + fmt.Printf("Assistant: %s \n", resp.Content) if resp.ResponseMeta != nil && resp.ResponseMeta.Usage != nil { - fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total)\n", + fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total) \n", resp.ResponseMeta.Usage.PromptTokens, resp.ResponseMeta.Usage.CompletionTokens, resp.ResponseMeta.Usage.TotalTokens) @@ -249,7 +243,7 @@ func main() { ``` -### generate_with_prefix +### 带前缀文本生成 ```go @@ -284,7 +278,7 @@ func main() { messages := []*schema.Message{ schema.UserMessage("Please write quick sort code"), - schema.AssistantMessage("```python\n", nil), + schema.AssistantMessage("```python \n", nil), } deepseek.SetPrefix(messages[1]) @@ -297,7 +291,7 @@ func main() { if !ok { fmt.Printf("No reasoning content") } else { - fmt.Printf("Reasoning: %v\n", reasoningContent) + fmt.Printf("Reasoning: %v \n", reasoningContent) } fmt.Printf("Content: %v\n", result) @@ -305,8 +299,7 @@ func main() { ``` -### stream - +### 流式生成 ```go package main @@ -369,7 +362,7 @@ func main() { fmt.Printf("Content: %s\n", resp.Content) } if resp.ResponseMeta != nil && resp.ResponseMeta.Usage != nil { - fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total)\n", + fmt.Printf("Tokens used: %d (prompt) + %d (completion) = %d (total) \n", resp.ResponseMeta.Usage.PromptTokens, resp.ResponseMeta.Usage.CompletionTokens, resp.ResponseMeta.Usage.TotalTokens) @@ -379,7 +372,7 @@ func main() { ``` -### intent_tool +### 工具调用 ```go @@ -423,7 +416,7 @@ func main() { "email": {Type: "string", Desc: "user's email"}}), }, { Name: "user_salary", - Desc: "Retrieve the user's salary based on their name and email.\n", + Desc: "Retrieve the user's salary based on their name and email.", ParamsOneOf: schema.NewParamsOneOfByParams( map[string]*schema.ParameterInfo{ "name": {Type: "string", Desc: "user's name"}, @@ -443,7 +436,7 @@ func main() { if err != nil { log.Fatalf("Generate of deepseek failed, err=%v", err) } - fmt.Printf("output: \n%v", resp) + fmt.Printf("output:%v \n", resp) streamResp, err := cm.Stream(ctx, []*schema.Message{ { @@ -472,15 +465,14 @@ func main() { if err != nil { log.Fatalf("ConcatMessages of deepseek failed, err=%v", err) } - fmt.Printf("stream output: \n%v", resp) + fmt.Printf("stream output:%v \n", resp) } ``` - -## For More Details +## 更多信息 - [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) - [DeepSeek Documentation](https://api-docs.deepseek.com/api/create-chat-completion) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_gemini.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_gemini.md index ed59ff645cf..c9761a36204 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_gemini.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_gemini.md @@ -1,33 +1,34 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - gemini weight: 0 --- -A Google Gemini implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个针对 [Eino](https://github.com/cloudwego/eino) 的 Google Gemini 实现，实现了 `ToolCallingChatModel` 接口。这使得能够与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -## Features +## 特性 -- Implements `github.com/cloudwego/eino/components/model.Model` -- Easy integration with Eino's model system -- Configurable model parameters -- Support for chat completion -- Support for streaming responses -- Custom response parsing support -- Flexible model configuration +- 实现了 `github.com/cloudwego/eino/components/model.Model` +- 轻松与 Eino 的模型系统集成 +- 可配置的模型参数 +- 支持聊天补全 +- 支持流式响应 +- 支持自定义响应解析 +- 灵活的模型配置 +- 支持对生成的响应进行缓存 -## Installation +## 安装 ```bash go get github.com/cloudwego/eino-ext/components/model/gemini@latest ``` -## Quick start +## 快速开始 -Here's a quick example of how to use the Gemini model: +以下是如何使用 Gemini 模型的快速示例： ```go package main @@ -57,7 +58,7 @@ func main() { cm, err := gemini.NewChatModel(ctx, &gemini.Config{ Client: client, - Model: "gemini-2.5-flash", + Model: "gemini-1.5-flash", ThinkingConfig: &genai.ThinkingConfig{ IncludeThoughts: true, ThinkingBudget: nil, @@ -67,7 +68,7 @@ func main() { log.Fatalf("NewChatModel of gemini failed, err=%v", err) } - // If you are using a model that supports image understanding (e.g., gemini-2.5-flash-image-preview), + // If you are using a model that supports image understanding (e.g., gemini-1.5-flash-image-preview), // you can provide both image and text input like this: /* image, err := os.ReadFile("./path/to/your/image.jpg") @@ -117,9 +118,9 @@ func main() { } ``` -## Configuration +## 配置 -The model can be configured using the `gemini.Config` struct: +可以使用 `gemini.Config` 结构体配置模型： ```go type Config struct { @@ -167,14 +168,64 @@ type Config struct { // ResponseModalities specifies the modalities the model can return. // Optional. - ResponseModalities []GeminiResponseModality + ResponseModalities [] + + MediaResolution genai.MediaResolution + + // Cache controls prefix cache settings for the model. + // Optional. used to CreatePrefixCache for reused inputs. + Cache *CacheConfig +} + +// CacheConfig controls prefix cache settings for the model. +type CacheConfig struct { + // TTL specifies how long cached resources remain valid (now + TTL). + TTL time.Duration `json:"ttl,omitempty"` + // ExpireTime sets the absolute expiration timestamp for cached resources. + ExpireTime time.Time `json:"expireTime,omitempty"` } ``` +## 缓存 + +该组件支持两种缓存策略以提高延迟并减少 API 调用： + +- 显式缓存（前缀缓存）：从系统指令、工具和消息中构建可重用的上下文。使用 `CreatePrefixCache` 创建缓存，并在后续请求中使用 `gemini.WithCachedContentName(...)` 传递其名称。通过 `CacheConfig`（`TTL`、`ExpireTime`）配置 TTL 和绝对到期时间。当使用缓存内容时，请求会省略系统指令和工具，并依赖于缓存的前缀。 +- 隐式缓存：由 Gemini 自身管理。服务可能会自动重用先前的请求或响应。到期和重用由 Gemini 控制，无法配置。 + +下面的示例展示了如何创建前缀缓存并在后续调用中重用它。 +```go +toolInfoList := []*schema.ToolInfo{ + { + Name: "tool_a", + Desc: "desc", + ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{}), + }, +} +cacheInfo, _ := cm.CreatePrefixCache(ctx, []*schema.Message{ + { + Role: schema.System, + Content: `aaa`, + }, + { + Role: schema.User, + Content: `bbb`, + }, + }, model.WithTools(toolInfoList)) + + +msg, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "give a very short summary about this transcript", + }, + }, gemini.WithCachedContentName(cacheInfo.Name)) +``` + -## examples +## 示例 -### generate +### 文本生成 ```go @@ -234,7 +285,7 @@ func main() { ``` -### generate_with_image +### 多模态支持(图片理解) ```go @@ -309,7 +360,117 @@ func main() { ``` -### stream +### 携带前缀缓存文本生成 + +```go + +package main + +import ( + "context" + "encoding/base64" + "fmt" + "log" + "os" + + "github.com/bytedance/sonic" + "github.com/cloudwego/eino/components/model" + "github.com/cloudwego/eino/components/tool/utils" + "github.com/cloudwego/eino/schema" + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" +) + +func main() { + ctx := context.Background() + + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: os.Getenv("GEMINI_API_KEY"), + }) + if err != nil { + log.Fatalf("genai.NewClient failed: %v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Model: os.Getenv("GEMINI_MODEL"), + Client: client, + }) + if err != nil { + log.Fatalf("gemini.NewChatModel failed: %v", err) + } + + type toolCallInput struct { + Answer int `json:"answer" jsonschema_description:"the answer of the question"` + } + answerTool, err := utils.InferTool("answer_to_user", + "answer to user", + func(ctx context.Context, in *toolCallInput) (string, error) { + return fmt.Sprintf("answer: %v", in.Answer), nil + }) + if err != nil { + log.Fatalf("utils.InferTool failed: %v", err) + } + + info, err := answerTool.Info(ctx) + if err != nil { + log.Fatalf("get tool info failed: %v", err) + } + + // this file is from gemini cache usage example + fileData, err := os.ReadFile("./a11.test.txt") + if err != nil { + log.Fatalf("os.ReadFile failed: %v", err) + } + + txtFileBase64 := base64.StdEncoding.EncodeToString(fileData) + cacheInfo, err := cm.CreatePrefixCache(ctx, []*schema.Message{ + { + Role: schema.System, + Content: `You are an expert at analyzing transcripts. +answer the question with the tool "answer_to_user" +always include the start_time and end_time of the transcript in the output`, + }, + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + { + Type: schema.ChatMessagePartTypeFileURL, + File: &schema.MessageInputFile{ + MessagePartCommon: schema.MessagePartCommon{ + Base64Data: &txtFileBase64, + MIMEType: "text/plain", + }, + }, + }, + }, + }, + }, model.WithTools([]*schema.ToolInfo{info}), model.WithToolChoice(schema.ToolChoiceForced)) + if err != nil { + log.Fatalf("CreatePrefixCache failed: %v", err) + } + + data, _ := sonic.MarshalIndent(cacheInfo, "", " ") + log.Printf("prefix cache info:\n%v\n", string(data)) + + msg, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "give a very short summary about this transcript", + }, + }, gemini.WithCachedContentName(cacheInfo.Name), + model.WithTools([]*schema.ToolInfo{info}), + model.WithToolChoice(schema.ToolChoiceForced)) + if err != nil { + log.Fatalf("Generate failed: %v", err) + } + msgData, _ := sonic.MarshalIndent(msg, "", " ") + log.Printf("model output:\n%v\n", string(msgData)) +} + +``` + +### 流式生成 ```go @@ -384,7 +545,114 @@ func main() { ``` -### image_generate +### 工具调用 + +```go + +package main + +import ( + "context" + "fmt" + "log" + "os" + + "google.golang.org/genai" + + "github.com/cloudwego/eino-ext/components/model/gemini" + "github.com/cloudwego/eino/schema" +) + +func main() { + apiKey := os.Getenv("GEMINI_API_KEY") + modelName := os.Getenv("GEMINI_MODEL") + + ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ + APIKey: apiKey, + }) + if err != nil { + log.Fatalf("NewClient of gemini failed, err=%v", err) + } + + cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Client: client, + Model: modelName, + ThinkingConfig: &genai.ThinkingConfig{ + IncludeThoughts: true, + ThinkingBudget: nil, + }, + }) + if err != nil { + log.Fatalf("NewChatModel of gemini failed, err=%v", err) + } + err = cm.BindTools([]*schema.ToolInfo{ + { + Name: "book_recommender", + Desc: "Recommends books based on user preferences and provides purchase links", + ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{ + "genre": { + Type: "string", + Desc: "Preferred book genre", + Enum: []string{"fiction", "sci-fi", "mystery", "biography", "business"}, + }, + "max_pages": { + Type: "integer", + Desc: "Maximum page length (0 for no limit)", + }, + "min_rating": { + Type: "number", + Desc: "Minimum user rating (0-5 scale)", + }, + }), + }, + }) + if err != nil { + log.Fatalf("Bind tools error: %v", err) + } + + resp, err := cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "Recommend business books with minimum 4.3 rating and max 350 pages", + }, + }) + if err != nil { + log.Fatalf("Generate error: %v", err) + } + + if len(resp.ToolCalls) > 0 { + fmt.Printf("Function called: \n") + if len(resp.ReasoningContent) > 0 { + fmt.Printf("ReasoningContent: %s\n", resp.ReasoningContent) + } + fmt.Println("Name: ", resp.ToolCalls[0].Function.Name) + fmt.Printf("Arguments: %s\n", resp.ToolCalls[0].Function.Arguments) + } else { + log.Printf("Function called without tool calls: %s\n", resp.Content) + } + + resp, err = cm.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + Content: "Recommend business books with minimum 4.3 rating and max 350 pages", + }, + resp, + { + Role: schema.Tool, + ToolCallID: resp.ToolCalls[0].ID, + Content: "{\"book name\":\"Microeconomics for Managers\"}", + }, + }) + if err != nil { + log.Fatalf("Generate error: %v", err) + } + fmt.Printf("Function call final result: %s\n", resp.Content) +} + +``` + +### 图片生成 ```go @@ -466,7 +734,7 @@ func main() { ``` -### intent_tool +### React Agent 模式示例 ```go @@ -478,104 +746,100 @@ import ( "log" "os" + "github.com/bytedance/sonic" + "github.com/cloudwego/eino/adk" + "github.com/cloudwego/eino/components/tool" + "github.com/cloudwego/eino/components/tool/utils" + "github.com/cloudwego/eino/compose" + "github.com/cloudwego/eino/schema" "google.golang.org/genai" "github.com/cloudwego/eino-ext/components/model/gemini" - "github.com/cloudwego/eino/schema" ) func main() { - apiKey := os.Getenv("GEMINI_API_KEY") - modelName := os.Getenv("GEMINI_MODEL") - ctx := context.Background() + client, err := genai.NewClient(ctx, &genai.ClientConfig{ - APIKey: apiKey, + APIKey: os.Getenv("GEMINI_API_KEY"), }) if err != nil { - log.Fatalf("NewClient of gemini failed, err=%v", err) + log.Fatalf("genai.NewClient failed, err=%v", err) } cm, err := gemini.NewChatModel(ctx, &gemini.Config{ + Model: os.Getenv("GEMINI_MODEL"), Client: client, - Model: modelName, - ThinkingConfig: &genai.ThinkingConfig{ - IncludeThoughts: true, - ThinkingBudget: nil, - }, }) if err != nil { - log.Fatalf("NewChatModel of gemini failed, err=%v", err) + log.Fatalf("gemini.NewChatModel failed, err=%v", err) } - err = cm.BindTools([]*schema.ToolInfo{ - { - Name: "book_recommender", - Desc: "Recommends books based on user preferences and provides purchase links", - ParamsOneOf: schema.NewParamsOneOfByParams(map[string]*schema.ParameterInfo{ - "genre": { - Type: "string", - Desc: "Preferred book genre", - Enum: []string{"fiction", "sci-fi", "mystery", "biography", "business"}, - }, - "max_pages": { - Type: "integer", - Desc: "Maximum page length (0 for no limit)", - }, - "min_rating": { - Type: "number", - Desc: "Minimum user rating (0-5 scale)", + + type toolCallInput struct { + LastCount int `json:"last_count" jsonschema_description:"the last count"` + } + countsTool, err := utils.InferTool("count_tool_call", + "count the number of tool calls", + func(ctx context.Context, in *toolCallInput) (string, error) { + counts := in.LastCount + 1 + return fmt.Sprintf("tool call counts: %v", counts), nil + }) + if err != nil { + log.Fatalf("utils.InferTool failed, err=%v", err) + } + + agent, err := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{ + Name: "react_agent", + Description: "react_agent", + Instruction: `call count_tool_call 5 times, then say 'done'`, + Model: cm, + ToolsConfig: adk.ToolsConfig{ + ToolsNodeConfig: compose.ToolsNodeConfig{ + Tools: []tool.BaseTool{ + countsTool, }, - }), + }, }, }) if err != nil { - log.Fatalf("Bind tools error: %v", err) + log.Fatalf("adk.NewChatModelAgent failed, err=%v", err) } - resp, err := cm.Generate(ctx, []*schema.Message{ - { - Role: schema.User, - Content: "Recommend business books with minimum 4.3 rating and max 350 pages", + iter := agent.Run(ctx, &adk.AgentInput{ + Messages: []adk.Message{ + { + Role: schema.User, + Content: "start to count", + }, }, }) - if err != nil { - log.Fatalf("Generate error: %v", err) - } + idx := 0 + for { + event, ok := iter.Next() + if !ok { + break + } - if len(resp.ToolCalls) > 0 { - fmt.Printf("Function called: \n") - if len(resp.ReasoningContent) > 0 { - fmt.Printf("ReasoningContent: %s\n", resp.ReasoningContent) + if event.Err != nil { + log.Fatalf("agent.Run failed, err=%v", event.Err) } - fmt.Println("Name: ", resp.ToolCalls[0].Function.Name) - fmt.Printf("Arguments: %s\n", resp.ToolCalls[0].Function.Arguments) - } else { - log.Printf("Function called without tool calls: %s\n", resp.Content) - } - resp, err = cm.Generate(ctx, []*schema.Message{ - { - Role: schema.User, - Content: "Recommend business books with minimum 4.3 rating and max 350 pages", - }, - resp, - { - Role: schema.Tool, - ToolCallID: resp.ToolCalls[0].ID, - Content: "{\"book name\":\"Microeconomics for Managers\"}", - }, - }) - if err != nil { - log.Fatalf("Generate error: %v", err) + msg, err_ := event.Output.MessageOutput.GetMessage() + if err_ != nil { + log.Fatalf("GetMessage failed, err=%v", err_) + } + + idx++ + msgData, _ := sonic.MarshalIndent(msg, "", " ") + log.Printf("\nmessage %v:\n%v\n", idx, string(msgData)) } - fmt.Printf("Function call final result: %s\n", resp.Content) } ``` -## For More Details +## 更多信息 - [Eino Documentation](https://github.com/cloudwego/eino) - [Gemini API Documentation](https://ai.google.dev/api/generate-content?hl=zh-cn#v1beta.GenerateContentResponse) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_ollama.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_ollama.md index a776c302952..6747136744e 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_ollama.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_ollama.md @@ -1,33 +1,33 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - ollama weight: 0 --- -A Ollama model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个针对 [Eino](https://github.com/cloudwego/eino) 的 Ollama 模型实现，实现了 `ToolCallingChatModel` 接口。这使得能够与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -## Features +## 特性 -- Implements `github.com/cloudwego/eino/components/model.Model` -- Easy integration with Eino's model system -- Configurable model parameters -- Support for chat completion -- Support for streaming responses -- Custom response parsing support -- Flexible model configuration +- 实现了 `github.com/cloudwego/eino/components/model.Model` +- 轻松与 Eino 的模型系统集成 +- 可配置的模型参数 +- 支持聊天补全 +- 支持流式响应 +- 支持自定义响应解析 +- 灵活的模型配置 -## Installation +## 安装 ```bash go get github.com/cloudwego/eino-ext/components/model/ollama@latest ``` -## Quick Start +## 快速开始 -Here's a quick example of how to use the Ollama model: +以下是如何使用 Ollama 模型的快速示例： ```go package main @@ -71,9 +71,9 @@ func main() { ``` -## Configuration +## 配置 -The model can be configured using the `ollama.ChatModelConfig` struct: +可以使用 `ollama.ChatModelConfig` 结构体配置模型： ```go @@ -135,9 +135,9 @@ type ThinkValue struct { ``` -## examples +## 示例 -### generate +### 文本生成 ```go @@ -181,7 +181,7 @@ func main() { ``` -### generate_with_image +### 多模态支持(图片理解) ```go @@ -244,7 +244,7 @@ func of[T any](a T) *T { ``` -### stream +### 流式生成 ```go @@ -270,7 +270,7 @@ func main() { Model: modelName, }) if err != nil { - log.Printf("NewChatModel failed, err=%v", err) + log.Printf("NewChatModel failed, err=%v\n", err) return } @@ -306,7 +306,7 @@ func main() { ``` -### intent_tool +### 工具调用 ```go @@ -393,7 +393,7 @@ func main() { ``` -### thinking +### 开启Thinking模式 ```go @@ -441,9 +441,7 @@ func main() { ``` - - -## For More Details +## 更多信息 - [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) - [Ollama Documentation](https://ollama.readthedocs.io/api/#generate-a-chat-completion) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_openai.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_openai.md index 554181d5111..1b33df9797a 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_openai.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_openai.md @@ -1,15 +1,15 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - openai weight: 0 --- -A OpenAI model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个针对 [Eino](https://github.com/cloudwego/eino) 的 OpenAI 模型实现，实现了 `ToolCallingChatModel` 接口。这使得能够与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -## Features +## 特性 - Implements `github.com/cloudwego/eino/components/model.Model` - Easy integration with Eino's model system @@ -19,15 +19,15 @@ A OpenAI model implementation for [Eino](https://github.com/cloudwego/eino) that - Custom response parsing support - Flexible model configuration -## Installation +## 安装 ```bash go get github.com/cloudwego/eino-ext/components/model/openai@latest ``` -## Quick Start +## 快速开始 -Here's a quick example of how to use the OpenAI model: +以下是如何使用 OpenAI 模型的快速示例： ```go @@ -47,7 +47,7 @@ func main() { ctx := context.Background() chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ - // if you want to use Azure OpenAI Service, set these two field. + // 如果您想使用 Azure OpenAI 服务，请设置这两个字段。 // BaseURL: "https://{RESOURCE_NAME}.openai.azure.com", // ByAzure: true, // APIVersion: "2024-06-01", @@ -82,9 +82,9 @@ func main() { ``` -## Configuration +## 配置 -The model can be configured using the `openai.ChatModelConfig` struct: +可以使用 `openai.ChatModelConfig` 结构体配置模型： ```go @@ -162,8 +162,8 @@ Stop []string `json:"stop,omitempty"` // Optional. Default: 0 PresencePenalty *float32 `json:"presence_penalty,omitempty"` -// ResponseFormat specifies the format of the model's response -// Optional. Use for structured outputs +// ResponseFormat 指定模型响应的格式 +// 可选。用于结构化输出 ResponseFormat *ChatCompletionResponseFormat `json:"response_format,omitempty"` // Seed enables deterministic sampling for consistent outputs @@ -201,9 +201,9 @@ Audio *Audio `json:"audio,omitempty"` ``` -## examples +## 示例 -### generate +### 文本生成 ```go @@ -215,15 +215,16 @@ import ( "log" "os" - "github.com/cloudwego/eino-ext/components/model/openai" "github.com/cloudwego/eino/schema" + + "github.com/cloudwego/eino-ext/components/model/openai" ) func main() { ctx := context.Background() chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ - // if you want to use Azure OpenAI Service, set these two field. + // 如果您想使用 Azure OpenAI 服务，请设置这两个字段。 // BaseURL: "https://{RESOURCE_NAME}.openai.azure.com", // ByAzure: true, // APIVersion: "2024-06-01", @@ -257,7 +258,7 @@ func main() { ``` -### generate_with_image +### 多模态支持(图片理解) ```go @@ -327,7 +328,7 @@ func of[T any](a T) *T { ``` -### stream +### 流式生成 ```go @@ -392,72 +393,7 @@ func main() { ``` -### audio_generate - -```go - -package main - -import ( - "context" - - "log" - "os" - - "github.com/bytedance/sonic" - "github.com/cloudwego/eino-ext/components/model/openai" - "github.com/cloudwego/eino/schema" -) - -func main() { - ctx := context.Background() - - chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ - // if you want to use Azure OpenAI Service, set these two field. - // BaseURL: "https://{RESOURCE_NAME}.openai.azure.com", - // ByAzure: true, - // APIVersion: "2024-06-01", - APIKey: os.Getenv("OPENAI_API_KEY"), - Model: os.Getenv("OPENAI_MODEL"), - BaseURL: os.Getenv("OPENAI_BASE_URL"), - ByAzure: func() bool { - if os.Getenv("OPENAI_BY_AZURE") == "true" { - return true - } - return false - }(), - ReasoningEffort: openai.ReasoningEffortLevelHigh, - Modalities: []openai.Modality{openai.AudioModality, openai.TextModality}, - Audio: &openai.Audio{ - Format: openai.AudioFormatMp3, - Voice: openai.AudioVoiceAlloy, - }, - }) - if err != nil { - log.Fatalf("NewChatModel failed, err=%v", err) - } - - resp, err := chatModel.Generate(ctx, []*schema.Message{ - { - Role: schema.User, - UserInputMultiContent: []schema.MessageInputPart{ - {Type: schema.ChatMessagePartTypeText, Text: "help me convert the following text to speech"}, - {Type: schema.ChatMessagePartTypeText, Text: "Hello, what can I help you with?"}, - }, - }, - }) - if err != nil { - log.Fatalf("Generate failed, err=%v", err) - } - - respBody, _ := sonic.MarshalIndent(resp, " ", " ") - log.Printf(" body: %s\n", string(respBody)) - -} - -``` - -### intent_tool +### 工具调用 ```go @@ -555,7 +491,72 @@ func main() { ``` -### structured +### 音频生成 + +```go + +package main + +import ( + "context" + + "log" + "os" + + "github.com/bytedance/sonic" + "github.com/cloudwego/eino-ext/components/model/openai" + "github.com/cloudwego/eino/schema" +) + +func main() { + ctx := context.Background() + + chatModel, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{ + // 如果您想使用 Azure OpenAI 服务，请设置这两个字段。 + // BaseURL: "https://{RESOURCE_NAME}.openai.azure.com", + // ByAzure: true, + // APIVersion: "2024-06-01", + APIKey: os.Getenv("OPENAI_API_KEY"), + Model: os.Getenv("OPENAI_MODEL"), + BaseURL: os.Getenv("OPENAI_BASE_URL"), + ByAzure: func() bool { + if os.Getenv("OPENAI_BY_AZURE") == "true" { + return true + } + return false + }(), + ReasoningEffort: openai.ReasoningEffortLevelHigh, + Modalities: []openai.Modality{openai.AudioModality, openai.TextModality}, + Audio: &openai.Audio{ + Format: openai.AudioFormatMp3, + Voice: openai.AudioVoiceAlloy, + }, + }) + if err != nil { + log.Fatalf("NewChatModel failed, err=%v", err) + } + + resp, err := chatModel.Generate(ctx, []*schema.Message{ + { + Role: schema.User, + UserInputMultiContent: []schema.MessageInputPart{ + {Type: schema.ChatMessagePartTypeText, Text: "help me convert the following text to speech"}, + {Type: schema.ChatMessagePartTypeText, Text: "Hello, what can I help you with?"}, + }, + }, + }) + if err != nil { + log.Fatalf("Generate failed, err=%v", err) + } + + respBody, _ := sonic.MarshalIndent(resp, " ", " ") + log.Printf(" body: %s\n", string(respBody)) + +} + +``` + +### 结构化输出 ```go @@ -661,7 +662,7 @@ func main() { -## For More Details +## 更多信息 - [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) - [OpenAI Documentation](https://platform.openai.com/docs/api-reference/chat/create) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_qianfan.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_qianfan.md index 9c59152a63c..19c6b2c5063 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_qianfan.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_qianfan.md @@ -1,33 +1,33 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - qianfan weight: 0 --- -A Qianfan model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个针对 [Eino](https://github.com/cloudwego/eino) 的千帆模型实现，实现了 `ToolCallingChatModel` 接口。这使得能够与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -## Features +## 特性 -- Implements `github.com/cloudwego/eino/components/model.Model` -- Easy integration with Eino's model system -- Configurable model parameters -- Support for chat completion -- Support for streaming responses -- Custom response parsing support -- Flexible model configuration +- 实现了 `github.com/cloudwego/eino/components/model.Model` +- 轻松与 Eino 的模型系统集成 +- 可配置的模型参数 +- 支持聊天补全 +- 支持流式响应 +- 自定义响应解析支持 +- 灵活的模型配置 -## Installation +## 安装 ```bash go get github.com/cloudwego/eino-ext/components/model/qianfan@latest ``` -## Quick Start +## 快速开始 -Here's a quick example of how to use the Qianfan model: +以下是如何使用千帆模型的快速示例： ```go @@ -80,9 +80,9 @@ func of[T any](t T) *T { } ``` -## Configuration +## 配置 -The model can be configured using the `qianfan.ChatModelConfig` struct: +可以使用 `qianfan.ChatModelConfig` 结构体配置模型： ```go @@ -134,7 +134,6 @@ type ChatModelConfig struct { // ResponseFormat specifies the format of the response. ResponseFormat *ResponseFormat } - ``` @@ -144,9 +143,9 @@ type ChatModelConfig struct { -## examples +## 示例 -### generate +### 文本生成 ```go @@ -199,7 +198,7 @@ func of[T any](t T) *T { ``` -### generate_with_image +### 多模态理解(图片理解) ```go @@ -274,7 +273,7 @@ func of[T any](t T) *T { ``` -### stream +### 流式生成 ```go @@ -344,7 +343,7 @@ func of[T any](t T) *T { ``` -### tool +### 工具调用 ```go @@ -440,7 +439,7 @@ func of[T any](t T) *T { -## For More Details +## 更多信息 -- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) -- [Qianfan Documentation](https://cloud.baidu.com/doc/qianfan-api/s/rm7u7qdiq) +- [Eino 文档](https://www.cloudwego.io/zh/docs/eino/) +- [千帆文档](https://cloud.baidu.com/doc/qianfan-api/s/rm7u7qdiq) diff --git a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_qwen.md b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_qwen.md index c7c01fb522b..d7d2faf0c1b 100644 --- a/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_qwen.md +++ b/content/zh/docs/eino/ecosystem_integration/chat_model/chat_model_qwen.md @@ -1,33 +1,33 @@ --- Description: "" -date: "2025-10-22" +date: "2025-12-02" lastmod: "" tags: [] title: ChatModel - qwen weight: 0 --- -A Qwen model implementation for [Eino](https://github.com/cloudwego/eino) that implements the `ToolCallingChatModel` interface. This enables seamless integration with Eino's LLM capabilities for enhanced natural language processing and generation. +一个针对 [Eino](https://github.com/cloudwego/eino) 的 Qwen 模型实现，实现了 `ToolCallingChatModel` 接口。这使得能够与 Eino 的 LLM 功能无缝集成，以增强自然语言处理和生成能力。 -## Features +## 特性 -- Implements `github.com/cloudwego/eino/components/model.Model` -- Easy integration with Eino's model system -- Configurable model parameters -- Support for chat completion -- Support for streaming responses -- Custom response parsing support -- Flexible model configuration +- 实现了 `github.com/cloudwego/eino/components/model.Model` +- 轻松与 Eino 的模型系统集成 +- 可配置的模型参数 +- 支持聊天补全 +- 支持流式响应 +- 自定义响应解析支持 +- 灵活的模型配置 -## Installation +## 安装 ```bash go get github.com/cloudwego/eino-ext/components/model/qwen@latest ``` -## Quick Start +## 快速开始 -Here's a quick example of how to use the Qwen model: +以下是如何使用 Qwen 模型的快速示例： ```go package main @@ -77,10 +77,9 @@ func of[T any](t T) *T { } ``` -## Configuration - -The model can be configured using the `qwen.ChatModelConfig` struct: +## 配置 +可以使用 `qwen.ChatModelConfig` 结构体配置模型： ```go type ChatModelConfig struct { @@ -164,13 +163,9 @@ EnableThinking *bool `json:"enable_thinking,omitempty"` ``` +## 示例 - - - -## examples - -### generate +### 文本生成 ```go @@ -222,7 +217,7 @@ func of[T any](t T) *T { ``` -### generate_with_image +### 多模态理解(图片理解) ```go @@ -298,7 +293,7 @@ func of[T any](t T) *T { ``` -### stream +### 流式生成 ```go @@ -385,7 +380,7 @@ func of[T any](t T) *T { ``` -### tool +### 工具调用 ```go @@ -527,6 +522,6 @@ func of[T any](t T) *T { -## For More Details -- [Eino Documentation](https://www.cloudwego.io/zh/docs/eino/) -- [Qwen Documentation](https://help.aliyun.com/zh/model-studio/use-qwen-by-calling-api) +## 更多信息 +- [Eino 文档](https://www.cloudwego.io/zh/docs/eino/) +- [Qwen 文档](https://help.aliyun.com/zh/model-studio/use-qwen-by-calling-api)