Merge pull request #2187 from oracle-devrel/genai-streaming-chatbot

streaming-chatbot

Author: anshoracle

ai/generative-ai-service/streaming-chatbot/LICENSE

@@ -0,0 +1,35 @@
+Copyright (c) 2025 Oracle and/or its affiliates.
+
+The Universal Permissive License (UPL), Version 1.0
+
+Subject to the condition set forth below, permission is hereby granted to any
+person obtaining a copy of this software, associated documentation and/or data
+(collectively the "Software"), free of charge and under any and all copyright
+rights in the Software, and any and all patent rights owned or freely
+licensable by each licensor hereunder covering either (i) the unmodified
+Software as contributed to or provided by such licensor, or (ii) the Larger
+Works (as defined below), to deal in both
+
+(a) the Software, and
+(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+one is included with the Software (each a "Larger Work" to which the Software
+is contributed by such licensors),
+
+without restriction, including without limitation the rights to copy, create
+derivative works of, display, perform, and distribute the Software and make,
+use, sell, offer for sale, import, export, have made, and have sold the
+Software and the Larger Work(s), and to sublicense the foregoing rights on
+either these or other terms.
+
+This license is subject to the following condition:
+The above copyright notice and either this complete permission notice or at
+a minimum a reference to the UPL must be included in all copies or
+substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ai/generative-ai-service/streaming-chatbot/README.md

@@ -0,0 +1,84 @@
+# Streaming Chatbot
+This asset is a Python-based chatbot that utilizes Oracle Cloud Infrastructure (OCI) Generative AI services to provide real-time, streaming responses. It is designed to interact with Cohere models available within OCI's Generative AI service. Users interested in leveraging other models, such as those from Meta or Google, can refer to the OCI Generative AI documentation for guidance on integrating these alternatives.
+
+Reviewed: 22.10.2025
+
+# When to use this asset?
+
+This asset is suitable for developers and organizations aiming to integrate AI-driven conversational interfaces into their applications, particularly when:
+
+- **Real-Time Interaction**: Immediate, dynamic responses are required in a chat interface.
+- **Stateless Processing**: Each user input is processed independently, without context from prior exchanges.
+- **Scalable AI Solutions**: Utilizing OCI's Generative AI services to handle varying loads and complex queries.
+
+# How to use this asset?
+
+To set up and run the `genai-streaming-chatbot`, follow these steps:
+
+1. **Prerequisites**:
+
+   - **Python Environment**: Ensure Python 3.6 or later is installed.
+   - **OCI Account**: An active Oracle Cloud Infrastructure account with necessary permissions.
+   - **API Key**: Generate and configure an API key for authentication.
+
+2. **Installation**:
+
+   - **Virtual Environment**: It's recommended to use a virtual environment to manage dependencies.
+
+     ```bash
+     pip install virtualenv
+     virtualenv oci_env
+     source oci_env/bin/activate
+     ```
+
+   - **Install OCI SDK**:
+
+     ```bash
+     pip install oci
+     ```
+
+3. **Configuration**:
+
+   - **API Key Setup**: Generate an API key pair in the OCI Console and configure it in the `~/.oci/config` file. Ensure the private key has appropriate permissions.
+   
+   - **Configuration File**: Create a `config` file in the `~/.oci` directory with the necessary details:
+
+     ```ini
+     [DEFAULT]
+     user=ocid1.user.oc1..<unique_ID>
+     fingerprint=<your_fingerprint>
+     key_file=~/.oci/oci_api_key.pem
+     tenancy=ocid1.tenancy.oc1..<unique_ID>
+     region=us-ashburn-1
+     ```
+
+4. **Running the Chatbot**:
+
+   - **Script Setup**: Save the chatbot script to a file, e.g., `genai_streaming_chatbot.py`.
+
+   - **Execute the Script**:
+
+     ```bash
+     python genai_streaming_chatbot.py
+     ```
+
+   - **Interaction**: Engage with the chatbot by typing messages. Type 'exit' to end the session.
+
+**Note**: This chatbot processes each input independently and does not retain context from previous interactions. To implement a conversational chatbot that maintains context, you would need to manage the conversation history externally by storing previous user inputs and model responses, then appending this history to each new prompt.
+
+# Useful Links
+
+- [OCI Python SDK Documentation](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/pythonsdk.htm)
+  - Comprehensive guide on using the OCI Python SDK.
+- [OCI Generative AI Inference API Documentation](https://docs.oracle.com/en-us/iaas/tools/python/latest/api/generative_ai_inference.html)
+  - Detailed information on the Generative AI Inference API.
+- [OCI Generative AI Service](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm)
+  - Detailed information on OCI Generative AI Service 
+
+# License
+
+Copyright (c) 2025 Oracle and/or its affiliates.
+
+Licensed under the Universal Permissive License (UPL), Version 1.0.
+
+See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE.txt) for more details.

ai/generative-ai-service/streaming-chatbot/files/genai-streaming-chatbot.py

@@ -0,0 +1,74 @@
+import json
+import oci
+from oci.generative_ai_inference import GenerativeAiInferenceClient
+from oci.generative_ai_inference.models import OnDemandServingMode, ChatDetails, CohereChatRequest
+
+compartment_id = "ocid1.compartment.oc1..XXXXXXXXXXXXXXXXXXXXX"
+model_id = "ocid1.generativeaimodel.oc1.XXXXXXXXXXXXXXXXXXXXXX"
+endpoint = "https://inference.generativeai.XXXXXXXXXXXXXXXXXXX"
+
+CONFIG_PROFILE = "DEFAULT"
+
+config = oci.config.from_file('~/.oci/config', CONFIG_PROFILE)
+
+def get_llm_response(prompt):
+    client = GenerativeAiInferenceClient(
+        config=config,
+        service_endpoint=endpoint,
+        retry_strategy=oci.retry.NoneRetryStrategy(),
+        timeout=(10, 240)
+    )
+
+    chat_request = CohereChatRequest(
+        message=prompt,
+        max_tokens=600,
+        temperature=0.5,
+        frequency_penalty=0,
+        top_p=0.75,
+        top_k=0,
+        is_stream=True  # Enable streaming
+    )
+
+    chat_detail = ChatDetails(
+        serving_mode=OnDemandServingMode(model_id=model_id),
+        chat_request=chat_request,
+        compartment_id=compartment_id
+    )
+
+    try:
+        response = client.chat(chat_detail)
+
+        buffer = ""
+        for event in response.data.events():
+            if not event.data:
+                continue
+
+            buffer += event.data.strip()
+            try:
+                event_json = json.loads(buffer)
+                buffer = ""  # clear buffer if parse succeeded
+            except json.JSONDecodeError:
+                # wait for more chunks
+                continue
+
+            text_chunk = event_json.get("text")
+            if text_chunk:
+                yield text_chunk
+
+    except Exception as e:
+        yield f"\n[Error during streaming: {e}]\n"
+
+def main():
+    print("OCI GenAI chatbot. Type 'exit' to quit.")
+    while True:
+        user_input = input("You: ")
+        if user_input.strip().lower() == 'exit':
+            print("Goodbye!")
+            break
+        print("Assistant: ", end="", flush=True)
+        for token in get_llm_response(user_input):
+            print(token, end="", flush=True)
+        print()
+
+if __name__ == "__main__":
+    main()