How to update Realtime SessionConfiguration?

[TypeDefinition]

Hi there!

Currently, I am trying to update a RealtimeSession's SessionConfiguration using the UpdateSessionRequest message. However, when I create a new SessionConfiguration, it automatically creates a new ClientSecret as well, resulting in me getting a invalid_request_error with the parameter session.client_secret.

To fix this, I had to change the SessionConfiguration.ClientSecret property to be public settable, so that I can set it to the same ClientSecret as my old SessionConfiguration.

Is there a proper way for me to do this? Thank you.

[StephenHodgson]

The client secret value is not set able. It comes directly from the API

[TypeDefinition]

Hi, I have made the repro steps. You can either download the whole Unity project here or put this code into a MonoBehaviour.

using OpenAI;
using OpenAI.Models;
using OpenAI.Realtime;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using UnityEngine;

public class ReproSteps : MonoBehaviour {
    private OpenAIClient api = null;
    private RealtimeSession session = null;

    private float timer = 10.0f;
    private bool hasUpdated = false;

    private void Start() {
        // Create a session with an initial configuration.
        CreateSession();
    }

    private void Update() {
        // Now try to update the configuration after a few seconds.
        timer -= Time.deltaTime;
        if (timer < 0.0f && !hasUpdated) {
            UpdateSession();
            hasUpdated = true;
        }
    }

    private async void CreateSession() {
        Debug.Log("Creating session.");
        try {
            api = new OpenAIClient(GetOpenAIAuthentication()) { EnableDebug = true };
            session = await api.RealtimeEndpoint.CreateSessionAsync(GetSessionConfiguration(), destroyCancellationToken);
            await session.ReceiveUpdatesAsync<IServerEvent>(OnServerEvent, destroyCancellationToken);
        } catch (Exception e) {
            switch (e) {
                case TaskCanceledException: break;
                case OperationCanceledException: break;
                default: Debug.LogException(e); break;
            }
        } finally {
            session?.Dispose();
            session = null;
            Debug.Log("Session disposed.");
        }
    }

    private async void UpdateSession() {
        Debug.Log("Updating session configuration.");
        try {
            OpenAI.Realtime.SessionConfiguration newConfig = GetUpdatedSessionConfiguration();
            // newConfig.ClientSecret = session.Configuration.ClientSecret; // If we could do this, the error would go away.
            await session.SendAsync(new UpdateSessionRequest(newConfig), destroyCancellationToken);
        } catch (Exception e) {
            Debug.LogException(e);
        }
    }

    private OpenAIAuthentication GetOpenAIAuthentication() {
        // For now just hardcode this.
        const string apiKey = "Your API Key Here";
        const string orgKey = "Your Organisation Key Here";
        const string projKey = "Your Project Key Here";
        return new OpenAIAuthentication(apiKey, orgKey, projKey);
    }

    private OpenAI.Realtime.SessionConfiguration GetSessionConfiguration() {
        return new OpenAI.Realtime.SessionConfiguration(
            Model.GPT4oRealtime,
            modalities: Modality.Text,
            instructions: "You are a helpful AI assistant",
            inputAudioTranscriptionSettings: new OpenAI.Realtime.InputAudioTranscriptionSettings(Model.Transcribe_GPT_4o, language: "en"), // The settings we use to transcribe what the human says. Without this, the human's speech will not get transcibed. Apparently the language setting is fucking useless.
            turnDetectionSettings: new OpenAI.Realtime.ServerVAD(silenceDuration: 2000), // We want Server VAD so that the AI automatically detects when speech starts or ends.
            tools: new List<OpenAI.Tool>(),
            toolChoice: "auto");
    }

    private OpenAI.Realtime.SessionConfiguration GetUpdatedSessionConfiguration() {
        // Let's say we want to only update the instructions.
        return new OpenAI.Realtime.SessionConfiguration(instructions: "You are a fearsome dinosaur. Rawr!");
    }

    private void OnServerEvent(IServerEvent @event) {
        switch (@event) {
            case RealtimeEventError error: throw error;
            default: break;
        }
    }
}

[TypeDefinition]

This is the error:

Exception: [unknown_parameter] Unknown parameter: 'session.client_secret'. Type: invalid_request_error Parameter: session.client_secret
ReproSteps.OnServerEvent (OpenAI.Realtime.IServerEvent event) (at Assets/Scripts/ReproSteps.cs:86)
OpenAI.Realtime.RealtimeSession.ReceiveUpdatesAsync[T] (System.Action`1[T] sessionEvent, System.Threading.CancellationToken cancellationToken) (at ./Library/PackageCache/com.openai.unity@de3f3effa41c/Runtime/Realtime/RealtimeSession.cs:189)
UnityEngine.Debug:LogException(Exception)
OpenAI.Realtime.<ReceiveUpdatesAsync>d__31`1:MoveNext() (at ./Library/PackageCache/com.openai.unity@de3f3effa41c/Runtime/Realtime/RealtimeSession.cs:202)
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

[StephenHodgson]

Thanks I'll look into it.

[StephenHodgson]

I found the problem. We're creating a new client secret in the constructor, but in reality we should not be doing this for updates.
The API does not expect a client secret for the update event, so I'll remove it before sending.

The following is the smallest repro:

var configuration = new SessionConfiguration(Model.GPT4oRealtime);
session = await OpenAIClient.RealtimeEndpoint.CreateSessionAsync(configuration);
// edit session configuration
var newConfig = new SessionConfiguration(instructions: "You are a fearsome dinosaur. Rawr!");
await session.SendAsync(new UpdateSessionRequest(newConfig));

I added this to the unit tests as well.

I'll make the fixes in the next release:

• Realtime Session Configuration cannot be updated #427
• com.openai.unity 8.8.6 #428

[TypeDefinition]

Thanks!