From c7857169e8f82f7804b2394fe0286c9d321c5b15 Mon Sep 17 00:00:00 2001 From: alistair-openai Date: Thu, 29 May 2025 08:44:02 -0400 Subject: [PATCH 1/3] publish final draft --- examples/Data-intensive-Realtime-apps.ipynb | 1498 +++++++++++++++++++ 1 file changed, 1498 insertions(+) create mode 100644 examples/Data-intensive-Realtime-apps.ipynb diff --git a/examples/Data-intensive-Realtime-apps.ipynb b/examples/Data-intensive-Realtime-apps.ipynb new file mode 100644 index 0000000000..33ac3efae2 --- /dev/null +++ b/examples/Data-intensive-Realtime-apps.ipynb @@ -0,0 +1,1498 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Practical guide to data-intensive apps with the Realtime API\n", + "\n", + "This cookbook serves as a practical guide to help AI Engineers maximize the effectiveness of OpenAI's Realtime API, specifically when dealing with data-intensive function calls. We'll focus on scenarios common in speech-to-speech agents, where vast amounts of data must be handled smoothly and efficiently.\n", + "\n", + "This post won't cover the basics of setting up a Realtime API solution. By following this guide, you'll gain clear insights and actionable strategies to enhance the performance and reliability of your real-time conversational agents. It addresses specific challenges unique to handling large amounts of data in real-time conversational contexts." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### What is the Realtime API?\n", + "\n", + "Before we dive in, let’s quickly recap the API for those who are new. The OpenAI Realtime API is a recent offering that supports low-latency, multimodal interactions—such as speech-to-speech conversations and live transcription. Picture scenarios like real-time voice-based customer support or live movie transcriptions." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### What is a data-intensive function call?\n", + "\n", + "Agents need access to tools and relevant data to perform their tasks. For instance, a financial analyst agent might pull real-time market data. In many cases, services already exist in your environment that expose this information through APIs.\n", + "\n", + "Historically, APIs weren’t designed with agents in mind and often return large volumes of data, depending on the service. As engineers, we frequently wrap these APIs with function calls to accelerate agent development—which makes perfect sense. Why reinvent what already exists?\n", + "\n", + "If not carefully optimized, these data-intensive function calls can quickly overwhelm the Realtime API—leading to slow responses or even failures to process user requests." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Setting the stage\n", + "\n", + "Our example centers on an NBA Scouting Agent that calls multiple functions to deliver in-depth analysis of upcoming draft prospects. To demonstrate practical guidelines for Realtime API interactions, we use large, realistic payloads inspired by NBA draft prospects. Below, you’ll find a monolithic `searchDraftProspects` function defined in the Realtime session to set the stage." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```json\n", + "// \"Hey, pull up point guards projected in the top 10 in the 2025 draft\"\n", + "{\n", + " \"type\": \"session.update\",\n", + " \"session\": {\n", + " \"tools\": [\n", + " {\n", + " \"type\": \"function\",\n", + " \"name\": \"searchDraftProspects\",\n", + " \"description\": \"Search draft prospects for a given year e.g., Point Guard\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"sign\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The player position\",\n", + " \"enum\": [\n", + " \"Point Guard\",\n", + " \"Shooting Guard\",\n", + " \"Small Forward\",\n", + " \"Power Forward\",\n", + " \"Center\",\n", + " \"Any\"\n", + " ]\n", + " },\n", + " year: { type: \"number\", description: \"Draft year e.g., 2025\" },\n", + " mockDraftRanking: { type: \"number\", description: \"Predicted Draft Ranking\" },\n", + " },\n", + " \"required\": [\"position\", \"year\"]\n", + " }\n", + " }\n", + " ],\n", + " \"tool_choice\": \"auto\",\n", + " }\n", + "}\n", + "```\n", + "\n", + "The searchDraftProspects function call returns a hefty payload. The example’s structure and size are drawn from real-world scenarios we’ve encountered.\n", + "\n", + "```json\n", + "// Example Payload\n", + "{\n", + " \"status\": {\n", + " \"code\": 200,\n", + " \"message\": \"SUCCESS\"\n", + " },\n", + " \"found\": 4274,\n", + " \"offset\": 0,\n", + " \"limit\": 10,\n", + " \"data\": [\n", + " {\n", + " \"prospectId\": 10001,\n", + " \"data\": {\n", + " \"ProspectInfo\": {\n", + " \"league\": \"NCAA\",\n", + " \"collegeId\": 301,\n", + " \"isDraftEligible\": true,\n", + " \"Player\": {\n", + " \"personalDetails\": {\n", + " \"firstName\": \"Jalen\",\n", + " \"lastName\": \"Storm\",\n", + " \"dateOfBirth\": \"2003-01-15\",\n", + " \"nationality\": \"USA\"\n", + " },\n", + " \"physicalAttributes\": {\n", + " \"position\": \"PG\",\n", + " \"height\": {\n", + " \"feet\": 6,\n", + " \"inches\": 4\n", + " },\n", + " \"weightPounds\": 205\n", + " },\n", + " \"hometown\": {\n", + " \"city\": \"Springfield\",\n", + " \"state\": \"IL\"\n", + " }\n", + " },\n", + " \"TeamInfo\": {\n", + " \"collegeTeam\": \"Springfield Tigers\",\n", + " \"conference\": \"Big West\",\n", + " \"teamRanking\": 12,\n", + " \"coach\": {\n", + " \"coachId\": 987,\n", + " \"coachName\": \"Marcus Reed\",\n", + " \"experienceYears\": 10\n", + " }\n", + " }\n", + " },\n", + " \"Stats\": {\n", + " \"season\": \"2025\",\n", + " \"gamesPlayed\": 32,\n", + " \"minutesPerGame\": 34.5,\n", + " \"shooting\": {\n", + " \"FieldGoalPercentage\": 47.2,\n", + " \"ThreePointPercentage\": 39.1,\n", + " \"FreeThrowPercentage\": 85.6\n", + " },\n", + " \"averages\": {\n", + " \"points\": 21.3,\n", + " \"rebounds\": 4.1,\n", + " \"assists\": 6.8,\n", + " \"steals\": 1.7,\n", + " \"blocks\": 0.3\n", + " }\n", + " },\n", + " \"Scouting\": {\n", + " \"evaluations\": {\n", + " \"strengths\": [\"Court vision\", \"Clutch shooting\"],\n", + " \"areasForImprovement\": [\"Defensive consistency\"]\n", + " },\n", + " \"scouts\": [\n", + " {\n", + " \"scoutId\": 501,\n", + " \"name\": \"Greg Hamilton\",\n", + " \"organization\": \"National Scouting Bureau\"\n", + " }\n", + " ]\n", + " },\n", + " \"DraftProjection\": {\n", + " \"mockDraftRanking\": 5,\n", + " \"lotteryPickProbability\": 88,\n", + " \"historicalComparisons\": [\n", + " {\n", + " \"player\": \"Chris Paul\",\n", + " \"similarityPercentage\": 85\n", + " }\n", + " ]\n", + " },\n", + " \"Media\": {\n", + " \"highlightReelUrl\": \"https://example.com/highlights/jalen-storm\",\n", + " \"socialMedia\": {\n", + " \"twitter\": \"@jstorm23\",\n", + " \"instagram\": \"@jstorm23_ig\"\n", + " }\n", + " },\n", + " \"Agent\": {\n", + " \"agentName\": \"Rick Allen\",\n", + " \"agency\": \"Elite Sports Management\",\n", + " \"contact\": {\n", + " \"email\": \"rallen@elitesports.com\",\n", + " \"phone\": \"555-123-4567\"\n", + " }\n", + " }\n", + " }\n", + " },\n", + " // ... Many thousands of tokens later.\n", + " ]\n", + "}\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 1. Agentic Workflows\n", + "\n", + "GPT-4.1 is a great place to build agentic workflows. In model training we emphasized providing a diverse range of agentic problem-solving trajectories, and our agentic harness for the model achieves state-of-the-art performance for non-reasoning models on SWE-bench Verified, solving 55% of problems. \n", + "\n", + "\n", + "## System Prompt Reminders\n", + "\n", + "In order to fully utilize the agentic capabilities of GPT-4.1, we recommend including three key types of reminders in all agent prompts. The following prompts are optimized specifically for the agentic coding workflow, but can be easily modified for general agentic use cases.\n", + "\n", + "1. Persistence: this ensures the model understands it is entering a multi-message turn, and prevents it from prematurely yielding control back to the user. Our example is the following:\n", + "\n", + "```\n", + "You are an agent - please keep going until the user’s query is completely resolved, before ending your turn and yielding back to the user. Only terminate your turn when you are sure that the problem is solved.\n", + "```\n", + "\n", + "2. Tool-calling: this encourages the model to make full use of its tools, and reduces its likelihood of hallucinating or guessing an answer. Our example is the following:\n", + "\n", + "```\n", + "If you are not sure about file content or codebase structure pertaining to the user’s request, use your tools to read files and gather the relevant information: do NOT guess or make up an answer.\n", + "```\n", + "\n", + "3. Planning \\[optional\\]: if desired, this ensures the model explicitly plans and reflects upon each tool call in text, instead of completing the task by chaining together a series of only tool calls. Our example is the following:\n", + "\n", + "```\n", + "You MUST plan extensively before each function call, and reflect extensively on the outcomes of the previous function calls. DO NOT do this entire process by making function calls only, as this can impair your ability to solve the problem and think insightfully.\n", + "```\n", + "\n", + "GPT-4.1 is trained to respond very closely to both user instructions and system prompts in the agentic setting. The model adhered closely to these three simple instructions and increased our internal SWE-bench Verified score by close to 20% \\- so we highly encourage starting any agent prompt with clear reminders covering the three categories listed above. As a whole, we find that these three instructions transform the model from a chatbot-like state into a much more “eager” agent, driving the interaction forward autonomously and independently. \n", + "\n", + "## Tool Calls\n", + "\n", + "Compared to previous models, GPT-4.1 has undergone more training on effectively utilizing tools passed as arguments in an OpenAI API request. We encourage developers to exclusively use the tools field to pass tools, rather than manually injecting tool descriptions into your prompt and writing a separate parser for tool calls, as some have reported doing in the past. This is the best way to minimize errors and ensure the model remains in distribution during tool-calling trajectories \\- in our own experiments, we observed a 2% increase in SWE-bench Verified pass rate when using API-parsed tool descriptions versus manually injecting the schemas into the system prompt.\n", + "\n", + "Developers should name tools clearly to indicate their purpose and add a clear, detailed description in the \"description\" field of the tool. Similarly, for each tool param, lean on good naming and descriptions to ensure appropriate usage. If your tool is particularly complicated and you'd like to provide examples of tool usage, we recommend that you create an `# Examples` section in your system prompt and place the examples there, rather than adding them into the \"description' field, which should remain thorough but relatively concise. Providing examples can be helpful to indicate when to use tools, whether to include user text alongside tool calls, and what parameters are appropriate for different inputs. Remember that you can use “Generate Anything” in the [Prompt Playground](https://platform.openai.com/playground) to get a good starting point for your new tool definitions.\n", + "\n", + "## Prompting-Induced Planning & Chain-of-Thought\n", + "\n", + "As mentioned already, developers can optionally prompt agents built with GPT-4.1 to plan and reflect between tool calls, instead of silently calling tools in an unbroken sequence. GPT-4.1 is not a reasoning model \\- meaning that it does not produce an internal chain of thought before answering \\- but in the prompt, a developer can induce the model to produce an explicit, step-by-step plan by using any variant of the Planning prompt component shown above. This can be thought of as the model “thinking out loud.” In our experimentation with the SWE-bench Verified agentic task, inducing explicit planning increased the pass rate by 4%. \n", + "\n", + "## Sample Prompt: SWE-bench Verified\n", + "\n", + "Below, we share the agentic prompt that we used to achieve our highest score on SWE-bench Verified, which features detailed instructions about workflow and problem-solving strategy. This general pattern can be used for any agentic task." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[{'id': 'msg_67fe92df26ac819182ffafce9ff4e4fc07c7e06242e51f8b',\n", + " 'content': [{'annotations': [],\n", + " 'text': \"Thank you for the report, but “Typerror” is too vague for me to start debugging right away.\\n\\n**To make progress, I need to:**\\n1. Find the exact error message text (e.g. `'TypeError: ...'`).\\n2. Find which file and which line/function/class the error occurred in.\\n3. Figure out what triggered the error (test file, usage, reproduction steps).\\n4. Find the root cause and details.\\n\\n**Next steps:**\\n- Investigate error/log/test output files for a Python `TypeError` message.\\n- Examine the relevant code sections for problematic type usage.\\n- If possible, reproduce the bug locally.\\n\\n**Plan:**\\n- First, I will search for test files and log output in the `/testbed` directory that may contain the full error message and stack trace.\\n\\nLet’s start by listing the contents of the `/testbed` directory to look for clues.\",\n", + " 'type': 'output_text'}],\n", + " 'role': 'assistant',\n", + " 'status': 'completed',\n", + " 'type': 'message'},\n", + " {'arguments': '{\"input\":\"!ls -l /testbed\"}',\n", + " 'call_id': 'call_frnxyJgKi5TsBem0nR9Zuzdw',\n", + " 'name': 'python',\n", + " 'type': 'function_call',\n", + " 'id': 'fc_67fe92e3da7081918fc18d5c96dddc1c07c7e06242e51f8b',\n", + " 'status': 'completed'}]" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from openai import OpenAI\n", + "import os\n", + "\n", + "client = OpenAI(\n", + " api_key=os.environ.get(\n", + " \"OPENAI_API_KEY\", \"\"\n", + " )\n", + ")\n", + "\n", + "SYS_PROMPT_SWEBENCH = \"\"\"\n", + "You will be tasked to fix an issue from an open-source repository.\n", + "\n", + "Your thinking should be thorough and so it's fine if it's very long. You can think step by step before and after each action you decide to take.\n", + "\n", + "You MUST iterate and keep going until the problem is solved.\n", + "\n", + "You already have everything you need to solve this problem in the /testbed folder, even without internet connection. I want you to fully solve this autonomously before coming back to me.\n", + "\n", + "Only terminate your turn when you are sure that the problem is solved. Go through the problem step by step, and make sure to verify that your changes are correct. NEVER end your turn without having solved the problem, and when you say you are going to make a tool call, make sure you ACTUALLY make the tool call, instead of ending your turn.\n", + "\n", + "THE PROBLEM CAN DEFINITELY BE SOLVED WITHOUT THE INTERNET.\n", + "\n", + "Take your time and think through every step - remember to check your solution rigorously and watch out for boundary cases, especially with the changes you made. Your solution must be perfect. If not, continue working on it. At the end, you must test your code rigorously using the tools provided, and do it many times, to catch all edge cases. If it is not robust, iterate more and make it perfect. Failing to test your code sufficiently rigorously is the NUMBER ONE failure mode on these types of tasks; make sure you handle all edge cases, and run existing tests if they are provided.\n", + "\n", + "You MUST plan extensively before each function call, and reflect extensively on the outcomes of the previous function calls. DO NOT do this entire process by making function calls only, as this can impair your ability to solve the problem and think insightfully.\n", + "\n", + "# Workflow\n", + "\n", + "## High-Level Problem Solving Strategy\n", + "\n", + "1. Understand the problem deeply. Carefully read the issue and think critically about what is required.\n", + "2. Investigate the codebase. Explore relevant files, search for key functions, and gather context.\n", + "3. Develop a clear, step-by-step plan. Break down the fix into manageable, incremental steps.\n", + "4. Implement the fix incrementally. Make small, testable code changes.\n", + "5. Debug as needed. Use debugging techniques to isolate and resolve issues.\n", + "6. Test frequently. Run tests after each change to verify correctness.\n", + "7. Iterate until the root cause is fixed and all tests pass.\n", + "8. Reflect and validate comprehensively. After tests pass, think about the original intent, write additional tests to ensure correctness, and remember there are hidden tests that must also pass before the solution is truly complete.\n", + "\n", + "Refer to the detailed sections below for more information on each step.\n", + "\n", + "## 1. Deeply Understand the Problem\n", + "Carefully read the issue and think hard about a plan to solve it before coding.\n", + "\n", + "## 2. Codebase Investigation\n", + "- Explore relevant files and directories.\n", + "- Search for key functions, classes, or variables related to the issue.\n", + "- Read and understand relevant code snippets.\n", + "- Identify the root cause of the problem.\n", + "- Validate and update your understanding continuously as you gather more context.\n", + "\n", + "## 3. Develop a Detailed Plan\n", + "- Outline a specific, simple, and verifiable sequence of steps to fix the problem.\n", + "- Break down the fix into small, incremental changes.\n", + "\n", + "## 4. Making Code Changes\n", + "- Before editing, always read the relevant file contents or section to ensure complete context.\n", + "- If a patch is not applied correctly, attempt to reapply it.\n", + "- Make small, testable, incremental changes that logically follow from your investigation and plan.\n", + "\n", + "## 5. Debugging\n", + "- Make code changes only if you have high confidence they can solve the problem\n", + "- When debugging, try to determine the root cause rather than addressing symptoms\n", + "- Debug for as long as needed to identify the root cause and identify a fix\n", + "- Use print statements, logs, or temporary code to inspect program state, including descriptive statements or error messages to understand what's happening\n", + "- To test hypotheses, you can also add test statements or functions\n", + "- Revisit your assumptions if unexpected behavior occurs.\n", + "\n", + "## 6. Testing\n", + "- Run tests frequently using `!python3 run_tests.py` (or equivalent).\n", + "- After each change, verify correctness by running relevant tests.\n", + "- If tests fail, analyze failures and revise your patch.\n", + "- Write additional tests if needed to capture important behaviors or edge cases.\n", + "- Ensure all tests pass before finalizing.\n", + "\n", + "## 7. Final Verification\n", + "- Confirm the root cause is fixed.\n", + "- Review your solution for logic correctness and robustness.\n", + "- Iterate until you are extremely confident the fix is complete and all tests pass.\n", + "\n", + "## 8. Final Reflection and Additional Testing\n", + "- Reflect carefully on the original intent of the user and the problem statement.\n", + "- Think about potential edge cases or scenarios that may not be covered by existing tests.\n", + "- Write additional tests that would need to pass to fully validate the correctness of your solution.\n", + "- Run these new tests and ensure they all pass.\n", + "- Be aware that there are additional hidden tests that must also pass for the solution to be successful.\n", + "- Do not assume the task is complete just because the visible tests pass; continue refining until you are confident the fix is robust and comprehensive.\n", + "\"\"\"\n", + "\n", + "PYTHON_TOOL_DESCRIPTION = \"\"\"This function is used to execute Python code or terminal commands in a stateful Jupyter notebook environment. python will respond with the output of the execution or time out after 60.0 seconds. Internet access for this session is disabled. Do not make external web requests or API calls as they will fail. Just as in a Jupyter notebook, you may also execute terminal commands by calling this function with a terminal command, prefaced with an exclamation mark.\n", + "\n", + "In addition, for the purposes of this task, you can call this function with an `apply_patch` command as input. `apply_patch` effectively allows you to execute a diff/patch against a file, but the format of the diff specification is unique to this task, so pay careful attention to these instructions. To use the `apply_patch` command, you should pass a message of the following structure as \"input\":\n", + "\n", + "%%bash\n", + "apply_patch <<\"EOF\"\n", + "*** Begin Patch\n", + "[YOUR_PATCH]\n", + "*** End Patch\n", + "EOF\n", + "\n", + "Where [YOUR_PATCH] is the actual content of your patch, specified in the following V4A diff format.\n", + "\n", + "*** [ACTION] File: [path/to/file] -> ACTION can be one of Add, Update, or Delete.\n", + "For each snippet of code that needs to be changed, repeat the following:\n", + "[context_before] -> See below for further instructions on context.\n", + "- [old_code] -> Precede the old code with a minus sign.\n", + "+ [new_code] -> Precede the new, replacement code with a plus sign.\n", + "[context_after] -> See below for further instructions on context.\n", + "\n", + "For instructions on [context_before] and [context_after]:\n", + "- By default, show 3 lines of code immediately above and 3 lines immediately below each change. If a change is within 3 lines of a previous change, do NOT duplicate the first change's [context_after] lines in the second change's [context_before] lines.\n", + "- If 3 lines of context is insufficient to uniquely identify the snippet of code within the file, use the @@ operator to indicate the class or function to which the snippet belongs. For instance, we might have:\n", + "@@ class BaseClass\n", + "[3 lines of pre-context]\n", + "- [old_code]\n", + "+ [new_code]\n", + "[3 lines of post-context]\n", + "\n", + "- If a code block is repeated so many times in a class or function such that even a single @@ statement and 3 lines of context cannot uniquely identify the snippet of code, you can use multiple `@@` statements to jump to the right context. For instance:\n", + "\n", + "@@ class BaseClass\n", + "@@ \tdef method():\n", + "[3 lines of pre-context]\n", + "- [old_code]\n", + "+ [new_code]\n", + "[3 lines of post-context]\n", + "\n", + "Note, then, that we do not use line numbers in this diff format, as the context is enough to uniquely identify code. An example of a message that you might pass as \"input\" to this function, in order to apply a patch, is shown below.\n", + "\n", + "%%bash\n", + "apply_patch <<\"EOF\"\n", + "*** Begin Patch\n", + "*** Update File: pygorithm/searching/binary_search.py\n", + "@@ class BaseClass\n", + "@@ def search():\n", + "- pass\n", + "+ raise NotImplementedError()\n", + "\n", + "@@ class Subclass\n", + "@@ def search():\n", + "- pass\n", + "+ raise NotImplementedError()\n", + "\n", + "*** End Patch\n", + "EOF\n", + "\n", + "File references can only be relative, NEVER ABSOLUTE. After the apply_patch command is run, python will always say \"Done!\", regardless of whether the patch was successfully applied or not. However, you can determine if there are issue and errors by looking at any warnings or logging lines printed BEFORE the \"Done!\" is output.\n", + "\"\"\"\n", + "\n", + "python_bash_patch_tool = {\n", + " \"type\": \"function\",\n", + " \"name\": \"python\",\n", + " \"description\": PYTHON_TOOL_DESCRIPTION,\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"strict\": True,\n", + " \"properties\": {\n", + " \"input\": {\n", + " \"type\": \"string\",\n", + " \"description\": \" The Python code, terminal command (prefaced by exclamation mark), or apply_patch command that you wish to execute.\",\n", + " }\n", + " },\n", + " \"required\": [\"input\"],\n", + " },\n", + "}\n", + "\n", + "# Additional harness setup:\n", + "# - Add your repo to /testbed\n", + "# - Add your issue to the first user message\n", + "# - Note: Even though we used a single tool for python, bash, and apply_patch, we generally recommend defining more granular tools that are focused on a single function\n", + "\n", + "response = client.responses.create(\n", + " instructions=SYS_PROMPT_SWEBENCH,\n", + " model=\"gpt-4.1-2025-04-14\",\n", + " tools=[python_bash_patch_tool],\n", + " input=f\"Please answer the following question:\\nBug: Typerror...\"\n", + ")\n", + "\n", + "response.to_dict()[\"output\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 2. Long context\n", + "\n", + "GPT-4.1 has a performant 1M token input context window, and is useful for a variety of long context tasks, including structured document parsing, re-ranking, selecting relevant information while ignoring irrelevant context, and performing multi-hop reasoning using context.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Optimal Context Size\n", + "\n", + "We observe very good performance on needle-in-a-haystack evaluations up to our full 1M token context, and we’ve observed very strong performance at complex tasks with a mix of both relevant and irrelevant code and other documents. However, long context performance can degrade as more items are required to be retrieved, or perform complex reasoning that requires knowledge of the state of the entire context (like performing a graph search, for example).\n", + "\n", + "## Tuning Context Reliance\n", + "\n", + "Consider the mix of external vs. internal world knowledge that might be required to answer your question. Sometimes it’s important for the model to use some of its own knowledge to connect concepts or make logical jumps, while in others it’s desirable to only use provided context\n", + "\n", + "```\n", + "# Instructions\n", + "// for internal knowledge\n", + "- Only use the documents in the provided External Context to answer the User Query. If you don't know the answer based on this context, you must respond \"I don't have the information needed to answer that\", even if a user insists on you answering the question.\n", + "// For internal and external knowledge\n", + "- By default, use the provided external context to answer the User Query, but if other basic knowledge is needed to answer, and you're confident in the answer, you can use some of your own knowledge to help answer the question.\n", + "```\n", + "\n", + "## Prompt Organization\n", + "\n", + "Especially in long context usage, placement of instructions and context can impact performance. If you have long context in your prompt, ideally place your instructions at both the beginning and end of the provided context, as we found this to perform better than only above or below. If you’d prefer to only have your instructions once, then above the provided context works better than below." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 3. Chain of Thought\n", + "\n", + "As mentioned above, GPT-4.1 is not a reasoning model, but prompting the model to think step by step (called “chain of thought”) can be an effective way for a model to break down problems into more manageable pieces, solve them, and improve overall output quality, with the tradeoff of higher cost and latency associated with using more output tokens. The model has been trained to perform well at agentic reasoning about and real-world problem solving, so it shouldn’t require much prompting to perform well.\n", + "\n", + "We recommend starting with this basic chain-of-thought instruction at the end of your prompt:\n", + "\n", + "```\n", + "...\n", + "\n", + "First, think carefully step by step about what documents are needed to answer the query. Then, print out the TITLE and ID of each document. Then, format the IDs into a list.\n", + "```\n", + "\n", + "From there, you should improve your chain-of-thought (CoT) prompt by auditing failures in your particular examples and evals, and addressing systematic planning and reasoning errors with more explicit instructions. In the unconstrained CoT prompt, there may be variance in the strategies it tries, and if you observe an approach that works well, you can codify that strategy in your prompt. Generally speaking, errors tend to occur from misunderstanding user intent, insufficient context gathering or analysis, or insufficient or incorrect step by step thinking, so watch out for these and try to address them with more opinionated instructions.\n", + "\n", + "Here is an example prompt instructing the model to focus more methodically on analyzing user intent and considering relevant context before proceeding to answer.\n", + "\n", + "```\n", + "# Reasoning Strategy\n", + "1. Query Analysis: Break down and analyze the query until you're confident about what it might be asking. Consider the provided context to help clarify any ambiguous or confusing information.\n", + "2. Context Analysis: Carefully select and analyze a large set of potentially relevant documents. Optimize for recall - it's okay if some are irrelevant, but the correct documents must be in this list, otherwise your final answer will be wrong. Analysis steps for each:\n", + "\ta. Analysis: An analysis of how it may or may not be relevant to answering the query.\n", + "\tb. Relevance rating: [high, medium, low, none]\n", + "3. Synthesis: summarize which documents are most relevant and why, including all documents with a relevance rating of medium or higher.\n", + "\n", + "# User Question\n", + "{user_question}\n", + "\n", + "# External Context\n", + "{external_context}\n", + "\n", + "First, think carefully step by step about what documents are needed to answer the query, closely adhering to the provided Reasoning Strategy. Then, print out the TITLE and ID of each document. Then, format the IDs into a list.\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 4. Instruction Following\n", + "\n", + "GPT-4.1 exhibits outstanding instruction-following performance, which developers can leverage to precisely shape and control the outputs for their particular use cases. Developers often extensively prompt for agentic reasoning steps, response tone and voice, tool calling information, output formatting, topics to avoid, and more. However, since the model follows instructions more literally, developers may need to include explicit specification around what to do or not to do. Furthermore, existing prompts optimized for other models may not immediately work with this model, because existing instructions are followed more closely and implicit rules are no longer being as strongly inferred." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Recommended Workflow\n", + "\n", + "Here is our recommended workflow for developing and debugging instructions in prompts:\n", + "\n", + "1. Start with an overall “Response Rules” or “Instructions” section with high-level guidance and bullet points. \n", + "2. If you’d like to change a more specific behavior, add a section to specify more details for that category, like `# Sample Phrases`. \n", + "3. If there are specific steps you’d like the model to follow in its workflow, add an ordered list and instruct the model to follow these steps.\n", + "4. If behavior still isn’t working as expected: \n", + " 1. Check for conflicting, underspecified, or wrong instructions and examples. If there are conflicting instructions, GPT-4.1 tends to follow the one closer to the end of the prompt.\n", + " 2. Add examples that demonstrate desired behavior; ensure that any important behavior demonstrated in your examples are also cited in your rules.\n", + " 3. It’s generally not necessary to use all-caps or other incentives like bribes or tips. We recommend starting without these, and only reaching for these if necessary for your particular prompt. Note that if your existing prompts include these techniques, it could cause GPT-4.1 to pay attention to it too strictly.\n", + "\n", + "*Note that using your preferred AI-powered IDE can be very helpful for iterating on prompts, including checking for consistency or conflicts, adding examples, or making cohesive updates like adding an instruction and updating instructions to demonstrate that instruction.*\n", + "\n", + "## Common Failure Modes\n", + "\n", + "These failure modes are not unique to GPT-4.1, but we share them here for general awareness and ease of debugging.\n", + "\n", + "* Instructing a model to always follow a specific behavior can occasionally induce adverse effects. For instance, if told “you must call a tool before responding to the user,” models may hallucinate tool inputs or call the tool with null values if they do not have enough information. Adding “if you don’t have enough information to call the tool, ask the user for the information you need” should mitigate this.\n", + "* When provided sample phrases, models can use those quotes verbatim and start to sound repetitive to users. Ensure you instruct the model to vary them as necessary.\n", + "* Without specific instructions, some models can be eager to provide additional prose to explain their decisions, or output more formatting in responses than may be desired. Provide instructions and potentially examples to help mitigate." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example Prompt: Customer Service\n", + "\n", + "This demonstrates best practices for a fictional customer service agent. Observe the diversity of rules, the specificity, the use of additional sections for greater detail, and an example to demonstrate precise behavior that incorporates all prior rules.\n", + "\n", + "Try running the following notebook cell - you should see both a user message and tool call, and the user message should start with a greeting, then echo back their answer, then mention they're about to call a tool. Try changing the instructions to shape the model behavior, or trying other user messages, to test instruction following performance." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[{'id': 'msg_67fe92d431548191b7ca6cd604b4784b06efc5beb16b3c5e',\n", + " 'content': [{'annotations': [],\n", + " 'text': \"Hi, you've reached NewTelco, how can I help you? 🌍✈️\\n\\nYou'd like to know the cost of international service while traveling to France. 🇫🇷 Let me check the latest details for you—one moment, please. 🕑\",\n", + " 'type': 'output_text'}],\n", + " 'role': 'assistant',\n", + " 'status': 'completed',\n", + " 'type': 'message'},\n", + " {'arguments': '{\"topic\":\"international service cost France\"}',\n", + " 'call_id': 'call_cF63DLeyhNhwfdyME3ZHd0yo',\n", + " 'name': 'lookup_policy_document',\n", + " 'type': 'function_call',\n", + " 'id': 'fc_67fe92d5d6888191b6cd7cf57f707e4606efc5beb16b3c5e',\n", + " 'status': 'completed'}]" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "SYS_PROMPT_CUSTOMER_SERVICE = \"\"\"You are a helpful customer service agent working for NewTelco, helping a user efficiently fulfill their request while adhering closely to provided guidelines.\n", + "\n", + "# Instructions\n", + "- Always greet the user with \"Hi, you've reached NewTelco, how can I help you?\"\n", + "- Always call a tool before answering factual questions about the company, its offerings or products, or a user's account. Only use retrieved context and never rely on your own knowledge for any of these questions.\n", + " - However, if you don't have enough information to properly call the tool, ask the user for the information you need.\n", + "- Escalate to a human if the user requests.\n", + "- Do not discuss prohibited topics (politics, religion, controversial current events, medical, legal, or financial advice, personal conversations, internal company operations, or criticism of any people or company).\n", + "- Rely on sample phrases whenever appropriate, but never repeat a sample phrase in the same conversation. Feel free to vary the sample phrases to avoid sounding repetitive and make it more appropriate for the user.\n", + "- Always follow the provided output format for new messages, including citations for any factual statements from retrieved policy documents.\n", + "- If you're going to call a tool, always message the user with an appropriate message before and after calling the tool.\n", + "- Maintain a professional and concise tone in all responses, and use emojis between sentences.\n", + "- If you've resolved the user's request, ask if there's anything else you can help with\n", + "\n", + "# Precise Response Steps (for each response)\n", + "1. If necessary, call tools to fulfill the user's desired action. Always message the user before and after calling a tool to keep them in the loop.\n", + "2. In your response to the user\n", + " a. Use active listening and echo back what you heard the user ask for.\n", + " b. Respond appropriately given the above guidelines.\n", + "\n", + "# Sample Phrases\n", + "## Deflecting a Prohibited Topic\n", + "- \"I'm sorry, but I'm unable to discuss that topic. Is there something else I can help you with?\"\n", + "- \"That's not something I'm able to provide information on, but I'm happy to help with any other questions you may have.\"\n", + "\n", + "## Before calling a tool\n", + "- \"To help you with that, I'll just need to verify your information.\"\n", + "- \"Let me check that for you—one moment, please.\"\n", + "- \"I'll retrieve the latest details for you now.\"\n", + "\n", + "## After calling a tool\n", + "- \"Okay, here's what I found: [response]\"\n", + "- \"So here's what I found: [response]\"\n", + "\n", + "# Output Format\n", + "- Always include your final response to the user.\n", + "- When providing factual information from retrieved context, always include citations immediately after the relevant statement(s). Use the following citation format:\n", + " - For a single source: [NAME](ID)\n", + " - For multiple sources: [NAME](ID), [NAME](ID)\n", + "- Only provide information about this company, its policies, its products, or the customer's account, and only if it is based on information provided in context. Do not answer questions outside this scope.\n", + "\n", + "# Example\n", + "## User\n", + "Can you tell me about your family plan options?\n", + "\n", + "## Assistant Response 1\n", + "### Message\n", + "\"Hi, you've reached NewTelco, how can I help you? 😊🎉\\n\\nYou'd like to know about our family plan options. 🤝 Let me check that for you—one moment, please. 🚀\"\n", + "\n", + "### Tool Calls\n", + "lookup_policy_document(topic=\"family plan options\")\n", + "\n", + "// After tool call, the assistant would follow up with:\n", + "\n", + "## Assistant Response 2 (after tool call)\n", + "### Message\n", + "\"Okay, here's what I found: 🎉 Our family plan allows up to 5 lines with shared data and a 10% discount for each additional line [Family Plan Policy](ID-010). 📱 Is there anything else I can help you with today? 😊\"\n", + "\"\"\"\n", + "\n", + "get_policy_doc = {\n", + " \"type\": \"function\",\n", + " \"name\": \"lookup_policy_document\",\n", + " \"description\": \"Tool to look up internal documents and policies by topic or keyword.\",\n", + " \"parameters\": {\n", + " \"strict\": True,\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"topic\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The topic or keyword to search for in company policies or documents.\",\n", + " },\n", + " },\n", + " \"required\": [\"topic\"],\n", + " \"additionalProperties\": False,\n", + " },\n", + "}\n", + "\n", + "get_user_acct = {\n", + " \"type\": \"function\",\n", + " \"name\": \"get_user_account_info\",\n", + " \"description\": \"Tool to get user account information\",\n", + " \"parameters\": {\n", + " \"strict\": True,\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"phone_number\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"Formatted as '(xxx) xxx-xxxx'\",\n", + " },\n", + " },\n", + " \"required\": [\"phone_number\"],\n", + " \"additionalProperties\": False,\n", + " },\n", + "}\n", + "\n", + "response = client.responses.create(\n", + " instructions=SYS_PROMPT_CUSTOMER_SERVICE,\n", + " model=\"gpt-4.1-2025-04-14\",\n", + " tools=[get_policy_doc, get_user_acct],\n", + " input=\"How much will it cost for international service? I'm traveling to France.\",\n", + " # input=\"Why was my last bill so high?\"\n", + ")\n", + "\n", + "response.to_dict()[\"output\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 5. General Advice\n", + "\n", + "## Prompt Structure\n", + "\n", + "For reference, here is a good starting point for structuring your prompts.\n", + "\n", + "```\n", + "# Role and Objective\n", + "\n", + "# Instructions\n", + "\n", + "## Sub-categories for more detailed instructions\n", + "\n", + "# Reasoning Steps\n", + "\n", + "# Output Format\n", + "\n", + "# Examples\n", + "## Example 1\n", + "\n", + "# Context\n", + "\n", + "# Final instructions and prompt to think step by step\n", + "```\n", + "\n", + "Add or remove sections to suit your needs, and experiment to determine what’s optimal for your usage.\n", + "\n", + "## Delimiters\n", + "\n", + "Here are some general guidelines for selecting the best delimiters for your prompt. Please refer to the Long Context section for special considerations for that context type.\n", + "\n", + "1. Markdown: We recommend starting here, and using markdown titles for major sections and subsections (including deeper hierarchy, to H4+). Use inline backticks or backtick blocks to precisely wrap code, and standard numbered or bulleted lists as needed. \n", + "2. XML: These also perform well, and we have improved adherence to information in XML with this model. XML is convenient to precisely wrap a section including start and end, add metadata to the tags for additional context, and enable nesting. Here is an example of using XML tags to nest examples in an example section, with inputs and outputs for each:\n", + "\n", + "```\n", + "\n", + "\n", + "San Francisco\n", + "- SF\n", + "\n", + "\n", + "```\n", + "\n", + "3. JSON is highly structured and well understood by the model particularly in coding contexts. However it can be more verbose, and require character escaping that can add overhead.\n", + "\n", + "Guidance specifically for adding a large number of documents or files to input context:\n", + "\n", + "* XML performed well in our long context testing. \n", + " * Example: `The quick brown fox jumps over the lazy dog` \n", + "* This format, proposed by Lee et al. ([ref](https://arxiv.org/pdf/2406.13121)), also performed well in our long context testing. \n", + " * Example: `ID: 1 | TITLE: The Fox | CONTENT: The quick brown fox jumps over the lazy dog` \n", + "* JSON performed particularly poorly. \n", + " * Example: `[{'id': 1, 'title': 'The Fox', 'content': 'The quick brown fox jumped over the lazy dog'}]`\n", + "\n", + "The model is trained to robustly understand structure in a variety of formats. Generally, use your judgement and think about what will provide clear information and “stand out” to the model. For example, if you’re retrieving documents that contain lots of XML, an XML-based delimiter will likely be less effective. \n", + "\n", + "## Caveats\n", + "\n", + "* In some isolated cases we have observed the model being resistant to producing very long, repetitive outputs, for example, analyzing hundreds of items one by one. If this is necessary for your use case, instruct the model strongly to output this information in full, and consider breaking down the problem or using a more concise approach. \n", + "* We have seen some rare instances of parallel tool calls being incorrect. We advise testing this, and considering setting the [parallel\\_tool\\_calls](https://platform.openai.com/docs/api-reference/responses/create#responses-create-parallel_tool_calls) param to false if you’re seeing issues." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Appendix: Generating and Applying File Diffs\n", + "\n", + "Developers have provided us feedback that accurate and well-formed diff generation is a critical capability to power coding-related tasks. To this end, the GPT-4.1 family features substantially improved diff capabilities relative to previous GPT models. Moreover, while GPT-4.1 has strong performance generating diffs of any format given clear instructions and examples, we open-source here one recommended diff format, on which the model has been extensively trained. We hope that in particular for developers just starting out, that this will take much of the guesswork out of creating diffs yourself. \n", + "\n", + "## Apply Patch\n", + "\n", + "See the example below for a prompt that applies our recommended tool call correctly." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "APPLY_PATCH_TOOL_DESC = \"\"\"This is a custom utility that makes it more convenient to add, remove, move, or edit code files. `apply_patch` effectively allows you to execute a diff/patch against a file, but the format of the diff specification is unique to this task, so pay careful attention to these instructions. To use the `apply_patch` command, you should pass a message of the following structure as \"input\":\n", + "\n", + "%%bash\n", + "apply_patch <<\"EOF\"\n", + "*** Begin Patch\n", + "[YOUR_PATCH]\n", + "*** End Patch\n", + "EOF\n", + "\n", + "Where [YOUR_PATCH] is the actual content of your patch, specified in the following V4A diff format.\n", + "\n", + "*** [ACTION] File: [path/to/file] -> ACTION can be one of Add, Update, or Delete.\n", + "For each snippet of code that needs to be changed, repeat the following:\n", + "[context_before] -> See below for further instructions on context.\n", + "- [old_code] -> Precede the old code with a minus sign.\n", + "+ [new_code] -> Precede the new, replacement code with a plus sign.\n", + "[context_after] -> See below for further instructions on context.\n", + "\n", + "For instructions on [context_before] and [context_after]:\n", + "- By default, show 3 lines of code immediately above and 3 lines immediately below each change. If a change is within 3 lines of a previous change, do NOT duplicate the first change’s [context_after] lines in the second change’s [context_before] lines.\n", + "- If 3 lines of context is insufficient to uniquely identify the snippet of code within the file, use the @@ operator to indicate the class or function to which the snippet belongs. For instance, we might have:\n", + "@@ class BaseClass\n", + "[3 lines of pre-context]\n", + "- [old_code]\n", + "+ [new_code]\n", + "[3 lines of post-context]\n", + "\n", + "- If a code block is repeated so many times in a class or function such that even a single @@ statement and 3 lines of context cannot uniquely identify the snippet of code, you can use multiple `@@` statements to jump to the right context. For instance:\n", + "\n", + "@@ class BaseClass\n", + "@@ \tdef method():\n", + "[3 lines of pre-context]\n", + "- [old_code]\n", + "+ [new_code]\n", + "[3 lines of post-context]\n", + "\n", + "Note, then, that we do not use line numbers in this diff format, as the context is enough to uniquely identify code. An example of a message that you might pass as \"input\" to this function, in order to apply a patch, is shown below.\n", + "\n", + "%%bash\n", + "apply_patch <<\"EOF\"\n", + "*** Begin Patch\n", + "*** Update File: pygorithm/searching/binary_search.py\n", + "@@ class BaseClass\n", + "@@ def search():\n", + "- pass\n", + "+ raise NotImplementedError()\n", + "\n", + "@@ class Subclass\n", + "@@ def search():\n", + "- pass\n", + "+ raise NotImplementedError()\n", + "\n", + "*** End Patch\n", + "EOF\n", + "\"\"\"\n", + "\n", + "APPLY_PATCH_TOOL = {\n", + " \"name\": \"apply_patch\",\n", + " \"description\": APPLY_PATCH_TOOL_DESC,\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"input\": {\n", + " \"type\": \"string\",\n", + " \"description\": \" The apply_patch command that you wish to execute.\",\n", + " }\n", + " },\n", + " \"required\": [\"input\"],\n", + " },\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Reference Implementation: apply\\_patch.py\n", + "\n", + "Here’s a reference implementation of the apply\\_patch tool that we used as part of model training. You’ll need to make this an executable and available as \\`apply\\_patch\\` from the shell where the model will execute commands:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "#!/usr/bin/env python3\n", + "\n", + "\"\"\"\n", + "A self-contained **pure-Python 3.9+** utility for applying human-readable\n", + "“pseudo-diff” patch files to a collection of text files.\n", + "\"\"\"\n", + "\n", + "from __future__ import annotations\n", + "\n", + "import pathlib\n", + "from dataclasses import dataclass, field\n", + "from enum import Enum\n", + "from typing import (\n", + " Callable,\n", + " Dict,\n", + " List,\n", + " Optional,\n", + " Tuple,\n", + " Union,\n", + ")\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# Domain objects\n", + "# --------------------------------------------------------------------------- #\n", + "class ActionType(str, Enum):\n", + " ADD = \"add\"\n", + " DELETE = \"delete\"\n", + " UPDATE = \"update\"\n", + "\n", + "\n", + "@dataclass\n", + "class FileChange:\n", + " type: ActionType\n", + " old_content: Optional[str] = None\n", + " new_content: Optional[str] = None\n", + " move_path: Optional[str] = None\n", + "\n", + "\n", + "@dataclass\n", + "class Commit:\n", + " changes: Dict[str, FileChange] = field(default_factory=dict)\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# Exceptions\n", + "# --------------------------------------------------------------------------- #\n", + "class DiffError(ValueError):\n", + " \"\"\"Any problem detected while parsing or applying a patch.\"\"\"\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# Helper dataclasses used while parsing patches\n", + "# --------------------------------------------------------------------------- #\n", + "@dataclass\n", + "class Chunk:\n", + " orig_index: int = -1\n", + " del_lines: List[str] = field(default_factory=list)\n", + " ins_lines: List[str] = field(default_factory=list)\n", + "\n", + "\n", + "@dataclass\n", + "class PatchAction:\n", + " type: ActionType\n", + " new_file: Optional[str] = None\n", + " chunks: List[Chunk] = field(default_factory=list)\n", + " move_path: Optional[str] = None\n", + "\n", + "\n", + "@dataclass\n", + "class Patch:\n", + " actions: Dict[str, PatchAction] = field(default_factory=dict)\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# Patch text parser\n", + "# --------------------------------------------------------------------------- #\n", + "@dataclass\n", + "class Parser:\n", + " current_files: Dict[str, str]\n", + " lines: List[str]\n", + " index: int = 0\n", + " patch: Patch = field(default_factory=Patch)\n", + " fuzz: int = 0\n", + "\n", + " # ------------- low-level helpers -------------------------------------- #\n", + " def _cur_line(self) -> str:\n", + " if self.index >= len(self.lines):\n", + " raise DiffError(\"Unexpected end of input while parsing patch\")\n", + " return self.lines[self.index]\n", + "\n", + " @staticmethod\n", + " def _norm(line: str) -> str:\n", + " \"\"\"Strip CR so comparisons work for both LF and CRLF input.\"\"\"\n", + " return line.rstrip(\"\\r\")\n", + "\n", + " # ------------- scanning convenience ----------------------------------- #\n", + " def is_done(self, prefixes: Optional[Tuple[str, ...]] = None) -> bool:\n", + " if self.index >= len(self.lines):\n", + " return True\n", + " if (\n", + " prefixes\n", + " and len(prefixes) > 0\n", + " and self._norm(self._cur_line()).startswith(prefixes)\n", + " ):\n", + " return True\n", + " return False\n", + "\n", + " def startswith(self, prefix: Union[str, Tuple[str, ...]]) -> bool:\n", + " return self._norm(self._cur_line()).startswith(prefix)\n", + "\n", + " def read_str(self, prefix: str) -> str:\n", + " \"\"\"\n", + " Consume the current line if it starts with *prefix* and return the text\n", + " **after** the prefix. Raises if prefix is empty.\n", + " \"\"\"\n", + " if prefix == \"\":\n", + " raise ValueError(\"read_str() requires a non-empty prefix\")\n", + " if self._norm(self._cur_line()).startswith(prefix):\n", + " text = self._cur_line()[len(prefix) :]\n", + " self.index += 1\n", + " return text\n", + " return \"\"\n", + "\n", + " def read_line(self) -> str:\n", + " \"\"\"Return the current raw line and advance.\"\"\"\n", + " line = self._cur_line()\n", + " self.index += 1\n", + " return line\n", + "\n", + " # ------------- public entry point -------------------------------------- #\n", + " def parse(self) -> None:\n", + " while not self.is_done((\"*** End Patch\",)):\n", + " # ---------- UPDATE ---------- #\n", + " path = self.read_str(\"*** Update File: \")\n", + " if path:\n", + " if path in self.patch.actions:\n", + " raise DiffError(f\"Duplicate update for file: {path}\")\n", + " move_to = self.read_str(\"*** Move to: \")\n", + " if path not in self.current_files:\n", + " raise DiffError(f\"Update File Error - missing file: {path}\")\n", + " text = self.current_files[path]\n", + " action = self._parse_update_file(text)\n", + " action.move_path = move_to or None\n", + " self.patch.actions[path] = action\n", + " continue\n", + "\n", + " # ---------- DELETE ---------- #\n", + " path = self.read_str(\"*** Delete File: \")\n", + " if path:\n", + " if path in self.patch.actions:\n", + " raise DiffError(f\"Duplicate delete for file: {path}\")\n", + " if path not in self.current_files:\n", + " raise DiffError(f\"Delete File Error - missing file: {path}\")\n", + " self.patch.actions[path] = PatchAction(type=ActionType.DELETE)\n", + " continue\n", + "\n", + " # ---------- ADD ---------- #\n", + " path = self.read_str(\"*** Add File: \")\n", + " if path:\n", + " if path in self.patch.actions:\n", + " raise DiffError(f\"Duplicate add for file: {path}\")\n", + " if path in self.current_files:\n", + " raise DiffError(f\"Add File Error - file already exists: {path}\")\n", + " self.patch.actions[path] = self._parse_add_file()\n", + " continue\n", + "\n", + " raise DiffError(f\"Unknown line while parsing: {self._cur_line()}\")\n", + "\n", + " if not self.startswith(\"*** End Patch\"):\n", + " raise DiffError(\"Missing *** End Patch sentinel\")\n", + " self.index += 1 # consume sentinel\n", + "\n", + " # ------------- section parsers ---------------------------------------- #\n", + " def _parse_update_file(self, text: str) -> PatchAction:\n", + " action = PatchAction(type=ActionType.UPDATE)\n", + " lines = text.split(\"\\n\")\n", + " index = 0\n", + " while not self.is_done(\n", + " (\n", + " \"*** End Patch\",\n", + " \"*** Update File:\",\n", + " \"*** Delete File:\",\n", + " \"*** Add File:\",\n", + " \"*** End of File\",\n", + " )\n", + " ):\n", + " def_str = self.read_str(\"@@ \")\n", + " section_str = \"\"\n", + " if not def_str and self._norm(self._cur_line()) == \"@@\":\n", + " section_str = self.read_line()\n", + "\n", + " if not (def_str or section_str or index == 0):\n", + " raise DiffError(f\"Invalid line in update section:\\n{self._cur_line()}\")\n", + "\n", + " if def_str.strip():\n", + " found = False\n", + " if def_str not in lines[:index]:\n", + " for i, s in enumerate(lines[index:], index):\n", + " if s == def_str:\n", + " index = i + 1\n", + " found = True\n", + " break\n", + " if not found and def_str.strip() not in [\n", + " s.strip() for s in lines[:index]\n", + " ]:\n", + " for i, s in enumerate(lines[index:], index):\n", + " if s.strip() == def_str.strip():\n", + " index = i + 1\n", + " self.fuzz += 1\n", + " found = True\n", + " break\n", + "\n", + " next_ctx, chunks, end_idx, eof = peek_next_section(self.lines, self.index)\n", + " new_index, fuzz = find_context(lines, next_ctx, index, eof)\n", + " if new_index == -1:\n", + " ctx_txt = \"\\n\".join(next_ctx)\n", + " raise DiffError(\n", + " f\"Invalid {'EOF ' if eof else ''}context at {index}:\\n{ctx_txt}\"\n", + " )\n", + " self.fuzz += fuzz\n", + " for ch in chunks:\n", + " ch.orig_index += new_index\n", + " action.chunks.append(ch)\n", + " index = new_index + len(next_ctx)\n", + " self.index = end_idx\n", + " return action\n", + "\n", + " def _parse_add_file(self) -> PatchAction:\n", + " lines: List[str] = []\n", + " while not self.is_done(\n", + " (\"*** End Patch\", \"*** Update File:\", \"*** Delete File:\", \"*** Add File:\")\n", + " ):\n", + " s = self.read_line()\n", + " if not s.startswith(\"+\"):\n", + " raise DiffError(f\"Invalid Add File line (missing '+'): {s}\")\n", + " lines.append(s[1:]) # strip leading '+'\n", + " return PatchAction(type=ActionType.ADD, new_file=\"\\n\".join(lines))\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# Helper functions\n", + "# --------------------------------------------------------------------------- #\n", + "def find_context_core(\n", + " lines: List[str], context: List[str], start: int\n", + ") -> Tuple[int, int]:\n", + " if not context:\n", + " return start, 0\n", + "\n", + " for i in range(start, len(lines)):\n", + " if lines[i : i + len(context)] == context:\n", + " return i, 0\n", + " for i in range(start, len(lines)):\n", + " if [s.rstrip() for s in lines[i : i + len(context)]] == [\n", + " s.rstrip() for s in context\n", + " ]:\n", + " return i, 1\n", + " for i in range(start, len(lines)):\n", + " if [s.strip() for s in lines[i : i + len(context)]] == [\n", + " s.strip() for s in context\n", + " ]:\n", + " return i, 100\n", + " return -1, 0\n", + "\n", + "\n", + "def find_context(\n", + " lines: List[str], context: List[str], start: int, eof: bool\n", + ") -> Tuple[int, int]:\n", + " if eof:\n", + " new_index, fuzz = find_context_core(lines, context, len(lines) - len(context))\n", + " if new_index != -1:\n", + " return new_index, fuzz\n", + " new_index, fuzz = find_context_core(lines, context, start)\n", + " return new_index, fuzz + 10_000\n", + " return find_context_core(lines, context, start)\n", + "\n", + "\n", + "def peek_next_section(\n", + " lines: List[str], index: int\n", + ") -> Tuple[List[str], List[Chunk], int, bool]:\n", + " old: List[str] = []\n", + " del_lines: List[str] = []\n", + " ins_lines: List[str] = []\n", + " chunks: List[Chunk] = []\n", + " mode = \"keep\"\n", + " orig_index = index\n", + "\n", + " while index < len(lines):\n", + " s = lines[index]\n", + " if s.startswith(\n", + " (\n", + " \"@@\",\n", + " \"*** End Patch\",\n", + " \"*** Update File:\",\n", + " \"*** Delete File:\",\n", + " \"*** Add File:\",\n", + " \"*** End of File\",\n", + " )\n", + " ):\n", + " break\n", + " if s == \"***\":\n", + " break\n", + " if s.startswith(\"***\"):\n", + " raise DiffError(f\"Invalid Line: {s}\")\n", + " index += 1\n", + "\n", + " last_mode = mode\n", + " if s == \"\":\n", + " s = \" \"\n", + " if s[0] == \"+\":\n", + " mode = \"add\"\n", + " elif s[0] == \"-\":\n", + " mode = \"delete\"\n", + " elif s[0] == \" \":\n", + " mode = \"keep\"\n", + " else:\n", + " raise DiffError(f\"Invalid Line: {s}\")\n", + " s = s[1:]\n", + "\n", + " if mode == \"keep\" and last_mode != mode:\n", + " if ins_lines or del_lines:\n", + " chunks.append(\n", + " Chunk(\n", + " orig_index=len(old) - len(del_lines),\n", + " del_lines=del_lines,\n", + " ins_lines=ins_lines,\n", + " )\n", + " )\n", + " del_lines, ins_lines = [], []\n", + "\n", + " if mode == \"delete\":\n", + " del_lines.append(s)\n", + " old.append(s)\n", + " elif mode == \"add\":\n", + " ins_lines.append(s)\n", + " elif mode == \"keep\":\n", + " old.append(s)\n", + "\n", + " if ins_lines or del_lines:\n", + " chunks.append(\n", + " Chunk(\n", + " orig_index=len(old) - len(del_lines),\n", + " del_lines=del_lines,\n", + " ins_lines=ins_lines,\n", + " )\n", + " )\n", + "\n", + " if index < len(lines) and lines[index] == \"*** End of File\":\n", + " index += 1\n", + " return old, chunks, index, True\n", + "\n", + " if index == orig_index:\n", + " raise DiffError(\"Nothing in this section\")\n", + " return old, chunks, index, False\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# Patch → Commit and Commit application\n", + "# --------------------------------------------------------------------------- #\n", + "def _get_updated_file(text: str, action: PatchAction, path: str) -> str:\n", + " if action.type is not ActionType.UPDATE:\n", + " raise DiffError(\"_get_updated_file called with non-update action\")\n", + " orig_lines = text.split(\"\\n\")\n", + " dest_lines: List[str] = []\n", + " orig_index = 0\n", + "\n", + " for chunk in action.chunks:\n", + " if chunk.orig_index > len(orig_lines):\n", + " raise DiffError(\n", + " f\"{path}: chunk.orig_index {chunk.orig_index} exceeds file length\"\n", + " )\n", + " if orig_index > chunk.orig_index:\n", + " raise DiffError(\n", + " f\"{path}: overlapping chunks at {orig_index} > {chunk.orig_index}\"\n", + " )\n", + "\n", + " dest_lines.extend(orig_lines[orig_index : chunk.orig_index])\n", + " orig_index = chunk.orig_index\n", + "\n", + " dest_lines.extend(chunk.ins_lines)\n", + " orig_index += len(chunk.del_lines)\n", + "\n", + " dest_lines.extend(orig_lines[orig_index:])\n", + " return \"\\n\".join(dest_lines)\n", + "\n", + "\n", + "def patch_to_commit(patch: Patch, orig: Dict[str, str]) -> Commit:\n", + " commit = Commit()\n", + " for path, action in patch.actions.items():\n", + " if action.type is ActionType.DELETE:\n", + " commit.changes[path] = FileChange(\n", + " type=ActionType.DELETE, old_content=orig[path]\n", + " )\n", + " elif action.type is ActionType.ADD:\n", + " if action.new_file is None:\n", + " raise DiffError(\"ADD action without file content\")\n", + " commit.changes[path] = FileChange(\n", + " type=ActionType.ADD, new_content=action.new_file\n", + " )\n", + " elif action.type is ActionType.UPDATE:\n", + " new_content = _get_updated_file(orig[path], action, path)\n", + " commit.changes[path] = FileChange(\n", + " type=ActionType.UPDATE,\n", + " old_content=orig[path],\n", + " new_content=new_content,\n", + " move_path=action.move_path,\n", + " )\n", + " return commit\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# User-facing helpers\n", + "# --------------------------------------------------------------------------- #\n", + "def text_to_patch(text: str, orig: Dict[str, str]) -> Tuple[Patch, int]:\n", + " lines = text.splitlines() # preserves blank lines, no strip()\n", + " if (\n", + " len(lines) < 2\n", + " or not Parser._norm(lines[0]).startswith(\"*** Begin Patch\")\n", + " or Parser._norm(lines[-1]) != \"*** End Patch\"\n", + " ):\n", + " raise DiffError(\"Invalid patch text - missing sentinels\")\n", + "\n", + " parser = Parser(current_files=orig, lines=lines, index=1)\n", + " parser.parse()\n", + " return parser.patch, parser.fuzz\n", + "\n", + "\n", + "def identify_files_needed(text: str) -> List[str]:\n", + " lines = text.splitlines()\n", + " return [\n", + " line[len(\"*** Update File: \") :]\n", + " for line in lines\n", + " if line.startswith(\"*** Update File: \")\n", + " ] + [\n", + " line[len(\"*** Delete File: \") :]\n", + " for line in lines\n", + " if line.startswith(\"*** Delete File: \")\n", + " ]\n", + "\n", + "\n", + "def identify_files_added(text: str) -> List[str]:\n", + " lines = text.splitlines()\n", + " return [\n", + " line[len(\"*** Add File: \") :]\n", + " for line in lines\n", + " if line.startswith(\"*** Add File: \")\n", + " ]\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# File-system helpers\n", + "# --------------------------------------------------------------------------- #\n", + "def load_files(paths: List[str], open_fn: Callable[[str], str]) -> Dict[str, str]:\n", + " return {path: open_fn(path) for path in paths}\n", + "\n", + "\n", + "def apply_commit(\n", + " commit: Commit,\n", + " write_fn: Callable[[str, str], None],\n", + " remove_fn: Callable[[str], None],\n", + ") -> None:\n", + " for path, change in commit.changes.items():\n", + " if change.type is ActionType.DELETE:\n", + " remove_fn(path)\n", + " elif change.type is ActionType.ADD:\n", + " if change.new_content is None:\n", + " raise DiffError(f\"ADD change for {path} has no content\")\n", + " write_fn(path, change.new_content)\n", + " elif change.type is ActionType.UPDATE:\n", + " if change.new_content is None:\n", + " raise DiffError(f\"UPDATE change for {path} has no new content\")\n", + " target = change.move_path or path\n", + " write_fn(target, change.new_content)\n", + " if change.move_path:\n", + " remove_fn(path)\n", + "\n", + "\n", + "def process_patch(\n", + " text: str,\n", + " open_fn: Callable[[str], str],\n", + " write_fn: Callable[[str, str], None],\n", + " remove_fn: Callable[[str], None],\n", + ") -> str:\n", + " if not text.startswith(\"*** Begin Patch\"):\n", + " raise DiffError(\"Patch text must start with *** Begin Patch\")\n", + " paths = identify_files_needed(text)\n", + " orig = load_files(paths, open_fn)\n", + " patch, _fuzz = text_to_patch(text, orig)\n", + " commit = patch_to_commit(patch, orig)\n", + " apply_commit(commit, write_fn, remove_fn)\n", + " return \"Done!\"\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# Default FS helpers\n", + "# --------------------------------------------------------------------------- #\n", + "def open_file(path: str) -> str:\n", + " with open(path, \"rt\", encoding=\"utf-8\") as fh:\n", + " return fh.read()\n", + "\n", + "\n", + "def write_file(path: str, content: str) -> None:\n", + " target = pathlib.Path(path)\n", + " target.parent.mkdir(parents=True, exist_ok=True)\n", + " with target.open(\"wt\", encoding=\"utf-8\") as fh:\n", + " fh.write(content)\n", + "\n", + "\n", + "def remove_file(path: str) -> None:\n", + " pathlib.Path(path).unlink(missing_ok=True)\n", + "\n", + "\n", + "# --------------------------------------------------------------------------- #\n", + "# CLI entry-point\n", + "# --------------------------------------------------------------------------- #\n", + "def main() -> None:\n", + " import sys\n", + "\n", + " patch_text = sys.stdin.read()\n", + " if not patch_text:\n", + " print(\"Please pass patch text through stdin\", file=sys.stderr)\n", + " return\n", + " try:\n", + " result = process_patch(patch_text, open_file, write_file, remove_file)\n", + " except DiffError as exc:\n", + " print(exc, file=sys.stderr)\n", + " return\n", + " print(result)\n", + "\n", + "\n", + "if __name__ == \"__main__\":\n", + " main()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Other Effective Diff Formats\n", + "\n", + "If you want to try using a different diff format, we found in testing that the SEARCH/REPLACE diff format used in Aider’s polyglot benchmark, as well as a pseudo-XML format with no internal escaping, both had high success rates.\n", + "\n", + "These diff formats share two key aspects: (1) they do not use line numbers, and (2) they provide both the exact code to be replaced, and the exact code with which to replace it, with clear delimiters between the two." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "SEARCH_REPLACE_DIFF_EXAMPLE = \"\"\"\n", + "path/to/file.py\n", + "```\n", + ">>>>>>> SEARCH\n", + "def search():\n", + " pass\n", + "=======\n", + "def search():\n", + " raise NotImplementedError()\n", + "<<<<<<< REPLACE\n", + "\"\"\"\n", + "\n", + "PSEUDO_XML_DIFF_EXAMPLE = \"\"\"\n", + "\n", + "\n", + "path/to/file.py\n", + "\n", + "\n", + "def search():\n", + " pass\n", + "\n", + "\n", + "def search():\n", + " raise NotImplementedError()\n", + "\n", + "\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} From 857bba51f5ea266794c1d668e3af769ff93d73d9 Mon Sep 17 00:00:00 2001 From: alistair-openai Date: Thu, 29 May 2025 11:06:33 -0400 Subject: [PATCH 2/3] update context window size comment --- authors.yaml | 6 + examples/Data-intensive-Realtime-apps.ipynb | 1490 ++++--------------- registry.yaml | 13 + 3 files changed, 327 insertions(+), 1182 deletions(-) diff --git a/authors.yaml b/authors.yaml index dda91cc10f..b44ff27ccd 100644 --- a/authors.yaml +++ b/authors.yaml @@ -347,3 +347,9 @@ tompakeman-oai: website: "https://www.linkedin.com/in/tom-pakeman/" avatar: "https://avatars.githubusercontent.com/u/204937754" +alistair-openai: + name: "Alistair Gillespie" + website: "https://www.linkedin.com/in/alistair-gillespie/" + avatar: "https://avatars.githubusercontent.com/u/210626148" + + diff --git a/examples/Data-intensive-Realtime-apps.ipynb b/examples/Data-intensive-Realtime-apps.ipynb index 33ac3efae2..ed3753c0e5 100644 --- a/examples/Data-intensive-Realtime-apps.ipynb +++ b/examples/Data-intensive-Realtime-apps.ipynb @@ -8,7 +8,7 @@ "\n", "This cookbook serves as a practical guide to help AI Engineers maximize the effectiveness of OpenAI's Realtime API, specifically when dealing with data-intensive function calls. We'll focus on scenarios common in speech-to-speech agents, where vast amounts of data must be handled smoothly and efficiently.\n", "\n", - "This post won't cover the basics of setting up a Realtime API solution. By following this guide, you'll gain clear insights and actionable strategies to enhance the performance and reliability of your real-time conversational agents. It addresses specific challenges unique to handling large amounts of data in real-time conversational contexts." + "This post won't cover the basics of setting up a Realtime API solution. Instead, you'll gain clear insights and actionable strategies to enhance the performance and reliability of your real-time conversational agents. It addresses specific challenges unique to handling large amounts of data in real-time conversational contexts." ] }, { @@ -17,7 +17,7 @@ "source": [ "### What is the Realtime API?\n", "\n", - "Before we dive in, let’s quickly recap the API for those who are new. The OpenAI Realtime API is a recent offering that supports low-latency, multimodal interactions—such as speech-to-speech conversations and live transcription. Picture scenarios like real-time voice-based customer support or live movie transcriptions." + "Before we dive in, let’s quickly recap the API for those who are new. The OpenAI Realtime API is a recent offering that supports low-latency, multimodal interactions—such as speech-to-speech conversations and live transcription. Picture scenarios like real-time voice-based customer support or live movie transcriptions. " ] }, { @@ -197,1281 +197,407 @@ "```" ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, { "cell_type": "markdown", "metadata": {}, "source": [ - "# 1. Agentic Workflows\n", - "\n", - "GPT-4.1 is a great place to build agentic workflows. In model training we emphasized providing a diverse range of agentic problem-solving trajectories, and our agentic harness for the model achieves state-of-the-art performance for non-reasoning models on SWE-bench Verified, solving 55% of problems. \n", - "\n", - "\n", - "## System Prompt Reminders\n", - "\n", - "In order to fully utilize the agentic capabilities of GPT-4.1, we recommend including three key types of reminders in all agent prompts. The following prompts are optimized specifically for the agentic coding workflow, but can be easily modified for general agentic use cases.\n", - "\n", - "1. Persistence: this ensures the model understands it is entering a multi-message turn, and prevents it from prematurely yielding control back to the user. Our example is the following:\n", - "\n", - "```\n", - "You are an agent - please keep going until the user’s query is completely resolved, before ending your turn and yielding back to the user. Only terminate your turn when you are sure that the problem is solved.\n", - "```\n", - "\n", - "2. Tool-calling: this encourages the model to make full use of its tools, and reduces its likelihood of hallucinating or guessing an answer. Our example is the following:\n", - "\n", - "```\n", - "If you are not sure about file content or codebase structure pertaining to the user’s request, use your tools to read files and gather the relevant information: do NOT guess or make up an answer.\n", - "```\n", - "\n", - "3. Planning \\[optional\\]: if desired, this ensures the model explicitly plans and reflects upon each tool call in text, instead of completing the task by chaining together a series of only tool calls. Our example is the following:\n", - "\n", - "```\n", - "You MUST plan extensively before each function call, and reflect extensively on the outcomes of the previous function calls. DO NOT do this entire process by making function calls only, as this can impair your ability to solve the problem and think insightfully.\n", - "```\n", - "\n", - "GPT-4.1 is trained to respond very closely to both user instructions and system prompts in the agentic setting. The model adhered closely to these three simple instructions and increased our internal SWE-bench Verified score by close to 20% \\- so we highly encourage starting any agent prompt with clear reminders covering the three categories listed above. As a whole, we find that these three instructions transform the model from a chatbot-like state into a much more “eager” agent, driving the interaction forward autonomously and independently. \n", - "\n", - "## Tool Calls\n", - "\n", - "Compared to previous models, GPT-4.1 has undergone more training on effectively utilizing tools passed as arguments in an OpenAI API request. We encourage developers to exclusively use the tools field to pass tools, rather than manually injecting tool descriptions into your prompt and writing a separate parser for tool calls, as some have reported doing in the past. This is the best way to minimize errors and ensure the model remains in distribution during tool-calling trajectories \\- in our own experiments, we observed a 2% increase in SWE-bench Verified pass rate when using API-parsed tool descriptions versus manually injecting the schemas into the system prompt.\n", - "\n", - "Developers should name tools clearly to indicate their purpose and add a clear, detailed description in the \"description\" field of the tool. Similarly, for each tool param, lean on good naming and descriptions to ensure appropriate usage. If your tool is particularly complicated and you'd like to provide examples of tool usage, we recommend that you create an `# Examples` section in your system prompt and place the examples there, rather than adding them into the \"description' field, which should remain thorough but relatively concise. Providing examples can be helpful to indicate when to use tools, whether to include user text alongside tool calls, and what parameters are appropriate for different inputs. Remember that you can use “Generate Anything” in the [Prompt Playground](https://platform.openai.com/playground) to get a good starting point for your new tool definitions.\n", - "\n", - "## Prompting-Induced Planning & Chain-of-Thought\n", - "\n", - "As mentioned already, developers can optionally prompt agents built with GPT-4.1 to plan and reflect between tool calls, instead of silently calling tools in an unbroken sequence. GPT-4.1 is not a reasoning model \\- meaning that it does not produce an internal chain of thought before answering \\- but in the prompt, a developer can induce the model to produce an explicit, step-by-step plan by using any variant of the Planning prompt component shown above. This can be thought of as the model “thinking out loud.” In our experimentation with the SWE-bench Verified agentic task, inducing explicit planning increased the pass rate by 4%. \n", - "\n", - "## Sample Prompt: SWE-bench Verified\n", - "\n", - "Below, we share the agentic prompt that we used to achieve our highest score on SWE-bench Verified, which features detailed instructions about workflow and problem-solving strategy. This general pattern can be used for any agentic task." + "## Guiding principles" ] }, { - "cell_type": "code", - "execution_count": 7, + "cell_type": "markdown", "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[{'id': 'msg_67fe92df26ac819182ffafce9ff4e4fc07c7e06242e51f8b',\n", - " 'content': [{'annotations': [],\n", - " 'text': \"Thank you for the report, but “Typerror” is too vague for me to start debugging right away.\\n\\n**To make progress, I need to:**\\n1. Find the exact error message text (e.g. `'TypeError: ...'`).\\n2. Find which file and which line/function/class the error occurred in.\\n3. Figure out what triggered the error (test file, usage, reproduction steps).\\n4. Find the root cause and details.\\n\\n**Next steps:**\\n- Investigate error/log/test output files for a Python `TypeError` message.\\n- Examine the relevant code sections for problematic type usage.\\n- If possible, reproduce the bug locally.\\n\\n**Plan:**\\n- First, I will search for test files and log output in the `/testbed` directory that may contain the full error message and stack trace.\\n\\nLet’s start by listing the contents of the `/testbed` directory to look for clues.\",\n", - " 'type': 'output_text'}],\n", - " 'role': 'assistant',\n", - " 'status': 'completed',\n", - " 'type': 'message'},\n", - " {'arguments': '{\"input\":\"!ls -l /testbed\"}',\n", - " 'call_id': 'call_frnxyJgKi5TsBem0nR9Zuzdw',\n", - " 'name': 'python',\n", - " 'type': 'function_call',\n", - " 'id': 'fc_67fe92e3da7081918fc18d5c96dddc1c07c7e06242e51f8b',\n", - " 'status': 'completed'}]" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], "source": [ - "from openai import OpenAI\n", - "import os\n", - "\n", - "client = OpenAI(\n", - " api_key=os.environ.get(\n", - " \"OPENAI_API_KEY\", \"\"\n", - " )\n", - ")\n", - "\n", - "SYS_PROMPT_SWEBENCH = \"\"\"\n", - "You will be tasked to fix an issue from an open-source repository.\n", - "\n", - "Your thinking should be thorough and so it's fine if it's very long. You can think step by step before and after each action you decide to take.\n", - "\n", - "You MUST iterate and keep going until the problem is solved.\n", - "\n", - "You already have everything you need to solve this problem in the /testbed folder, even without internet connection. I want you to fully solve this autonomously before coming back to me.\n", - "\n", - "Only terminate your turn when you are sure that the problem is solved. Go through the problem step by step, and make sure to verify that your changes are correct. NEVER end your turn without having solved the problem, and when you say you are going to make a tool call, make sure you ACTUALLY make the tool call, instead of ending your turn.\n", - "\n", - "THE PROBLEM CAN DEFINITELY BE SOLVED WITHOUT THE INTERNET.\n", - "\n", - "Take your time and think through every step - remember to check your solution rigorously and watch out for boundary cases, especially with the changes you made. Your solution must be perfect. If not, continue working on it. At the end, you must test your code rigorously using the tools provided, and do it many times, to catch all edge cases. If it is not robust, iterate more and make it perfect. Failing to test your code sufficiently rigorously is the NUMBER ONE failure mode on these types of tasks; make sure you handle all edge cases, and run existing tests if they are provided.\n", - "\n", - "You MUST plan extensively before each function call, and reflect extensively on the outcomes of the previous function calls. DO NOT do this entire process by making function calls only, as this can impair your ability to solve the problem and think insightfully.\n", - "\n", - "# Workflow\n", - "\n", - "## High-Level Problem Solving Strategy\n", + "### 1. Break down unwieldy functions into smaller ones with clear roles and responsibilities\n", "\n", - "1. Understand the problem deeply. Carefully read the issue and think critically about what is required.\n", - "2. Investigate the codebase. Explore relevant files, search for key functions, and gather context.\n", - "3. Develop a clear, step-by-step plan. Break down the fix into manageable, incremental steps.\n", - "4. Implement the fix incrementally. Make small, testable code changes.\n", - "5. Debug as needed. Use debugging techniques to isolate and resolve issues.\n", - "6. Test frequently. Run tests after each change to verify correctness.\n", - "7. Iterate until the root cause is fixed and all tests pass.\n", - "8. Reflect and validate comprehensively. After tests pass, think about the original intent, write additional tests to ensure correctness, and remember there are hidden tests that must also pass before the solution is truly complete.\n", + "It almost goes without saying—when building function calls, your top priority is to design clear, well-defined functions. This makes it easy to trim response sizes and avoid overwhelming the model. Each function call should be straightforward to explain, sharply scoped, and return only the information needed for its purpose. Overlapping responsibilities between functions inevitably invites confusion.\n", "\n", - "Refer to the detailed sections below for more information on each step.\n", + "For example, we can limit the `searchDraftProspects` function call to return only general details—such as player stats—for each prospect, dramatically reducing the response size. If more information is needed, the new `getProspectDetails` function call provides expanded details. There’s no universal solution; the right approach depends on your use case and data model.\n", "\n", - "## 1. Deeply Understand the Problem\n", - "Carefully read the issue and think hard about a plan to solve it before coding.\n", - "\n", - "## 2. Codebase Investigation\n", - "- Explore relevant files and directories.\n", - "- Search for key functions, classes, or variables related to the issue.\n", - "- Read and understand relevant code snippets.\n", - "- Identify the root cause of the problem.\n", - "- Validate and update your understanding continuously as you gather more context.\n", - "\n", - "## 3. Develop a Detailed Plan\n", - "- Outline a specific, simple, and verifiable sequence of steps to fix the problem.\n", - "- Break down the fix into small, incremental changes.\n", - "\n", - "## 4. Making Code Changes\n", - "- Before editing, always read the relevant file contents or section to ensure complete context.\n", - "- If a patch is not applied correctly, attempt to reapply it.\n", - "- Make small, testable, incremental changes that logically follow from your investigation and plan.\n", - "\n", - "## 5. Debugging\n", - "- Make code changes only if you have high confidence they can solve the problem\n", - "- When debugging, try to determine the root cause rather than addressing symptoms\n", - "- Debug for as long as needed to identify the root cause and identify a fix\n", - "- Use print statements, logs, or temporary code to inspect program state, including descriptive statements or error messages to understand what's happening\n", - "- To test hypotheses, you can also add test statements or functions\n", - "- Revisit your assumptions if unexpected behavior occurs.\n", - "\n", - "## 6. Testing\n", - "- Run tests frequently using `!python3 run_tests.py` (or equivalent).\n", - "- After each change, verify correctness by running relevant tests.\n", - "- If tests fail, analyze failures and revise your patch.\n", - "- Write additional tests if needed to capture important behaviors or edge cases.\n", - "- Ensure all tests pass before finalizing.\n", - "\n", - "## 7. Final Verification\n", - "- Confirm the root cause is fixed.\n", - "- Review your solution for logic correctness and robustness.\n", - "- Iterate until you are extremely confident the fix is complete and all tests pass.\n", - "\n", - "## 8. Final Reflection and Additional Testing\n", - "- Reflect carefully on the original intent of the user and the problem statement.\n", - "- Think about potential edge cases or scenarios that may not be covered by existing tests.\n", - "- Write additional tests that would need to pass to fully validate the correctness of your solution.\n", - "- Run these new tests and ensure they all pass.\n", - "- Be aware that there are additional hidden tests that must also pass for the solution to be successful.\n", - "- Do not assume the task is complete just because the visible tests pass; continue refining until you are confident the fix is robust and comprehensive.\n", - "\"\"\"\n", - "\n", - "PYTHON_TOOL_DESCRIPTION = \"\"\"This function is used to execute Python code or terminal commands in a stateful Jupyter notebook environment. python will respond with the output of the execution or time out after 60.0 seconds. Internet access for this session is disabled. Do not make external web requests or API calls as they will fail. Just as in a Jupyter notebook, you may also execute terminal commands by calling this function with a terminal command, prefaced with an exclamation mark.\n", - "\n", - "In addition, for the purposes of this task, you can call this function with an `apply_patch` command as input. `apply_patch` effectively allows you to execute a diff/patch against a file, but the format of the diff specification is unique to this task, so pay careful attention to these instructions. To use the `apply_patch` command, you should pass a message of the following structure as \"input\":\n", - "\n", - "%%bash\n", - "apply_patch <<\"EOF\"\n", - "*** Begin Patch\n", - "[YOUR_PATCH]\n", - "*** End Patch\n", - "EOF\n", - "\n", - "Where [YOUR_PATCH] is the actual content of your patch, specified in the following V4A diff format.\n", - "\n", - "*** [ACTION] File: [path/to/file] -> ACTION can be one of Add, Update, or Delete.\n", - "For each snippet of code that needs to be changed, repeat the following:\n", - "[context_before] -> See below for further instructions on context.\n", - "- [old_code] -> Precede the old code with a minus sign.\n", - "+ [new_code] -> Precede the new, replacement code with a plus sign.\n", - "[context_after] -> See below for further instructions on context.\n", - "\n", - "For instructions on [context_before] and [context_after]:\n", - "- By default, show 3 lines of code immediately above and 3 lines immediately below each change. If a change is within 3 lines of a previous change, do NOT duplicate the first change's [context_after] lines in the second change's [context_before] lines.\n", - "- If 3 lines of context is insufficient to uniquely identify the snippet of code within the file, use the @@ operator to indicate the class or function to which the snippet belongs. For instance, we might have:\n", - "@@ class BaseClass\n", - "[3 lines of pre-context]\n", - "- [old_code]\n", - "+ [new_code]\n", - "[3 lines of post-context]\n", - "\n", - "- If a code block is repeated so many times in a class or function such that even a single @@ statement and 3 lines of context cannot uniquely identify the snippet of code, you can use multiple `@@` statements to jump to the right context. For instance:\n", - "\n", - "@@ class BaseClass\n", - "@@ \tdef method():\n", - "[3 lines of pre-context]\n", - "- [old_code]\n", - "+ [new_code]\n", - "[3 lines of post-context]\n", - "\n", - "Note, then, that we do not use line numbers in this diff format, as the context is enough to uniquely identify code. An example of a message that you might pass as \"input\" to this function, in order to apply a patch, is shown below.\n", - "\n", - "%%bash\n", - "apply_patch <<\"EOF\"\n", - "*** Begin Patch\n", - "*** Update File: pygorithm/searching/binary_search.py\n", - "@@ class BaseClass\n", - "@@ def search():\n", - "- pass\n", - "+ raise NotImplementedError()\n", - "\n", - "@@ class Subclass\n", - "@@ def search():\n", - "- pass\n", - "+ raise NotImplementedError()\n", - "\n", - "*** End Patch\n", - "EOF\n", - "\n", - "File references can only be relative, NEVER ABSOLUTE. After the apply_patch command is run, python will always say \"Done!\", regardless of whether the patch was successfully applied or not. However, you can determine if there are issue and errors by looking at any warnings or logging lines printed BEFORE the \"Done!\" is output.\n", - "\"\"\"\n", - "\n", - "python_bash_patch_tool = {\n", - " \"type\": \"function\",\n", - " \"name\": \"python\",\n", - " \"description\": PYTHON_TOOL_DESCRIPTION,\n", - " \"parameters\": {\n", - " \"type\": \"object\",\n", - " \"strict\": True,\n", - " \"properties\": {\n", - " \"input\": {\n", - " \"type\": \"string\",\n", - " \"description\": \" The Python code, terminal command (prefaced by exclamation mark), or apply_patch command that you wish to execute.\",\n", + "```json\n", + "{\n", + " \"tools\": [\n", + " {\n", + " \"type\": \"function\",\n", + " \"name\": \"searchDraftProspects\",\n", + " \"description\": \"Search NBA draft prospects by position, draft year, and projected ranking, returning only general statistics to optimize response size.\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"position\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The player's basketball position.\",\n", + " \"enum\": [\n", + " \"Point Guard\",\n", + " \"Shooting Guard\",\n", + " \"Small Forward\",\n", + " \"Power Forward\",\n", + " \"Center\",\n", + " \"Any\"\n", + " ]\n", + " },\n", + " \"year\": {\n", + " \"type\": \"number\",\n", + " \"description\": \"Draft year, e.g., 2025\"\n", + " },\n", + " \"maxMockDraftRanking\": {\n", + " \"type\": \"number\",\n", + " \"description\": \"Maximum predicted draft ranking (e.g., top 10)\"\n", " }\n", - " },\n", - " \"required\": [\"input\"],\n", - " },\n", + " },\n", + " \"required\": [\"position\", \"year\"]\n", + " }\n", + " },\n", + " {\n", + " \"type\": \"function\",\n", + " \"name\": \"getProspectDetails\",\n", + " \"description\": \"Fetch detailed information for a specific NBA prospect, including comprehensive stats, agent details, and scouting reports.\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"playerName\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"Full name of the prospect (e.g., Jalen Storm)\"\n", + " },\n", + " \"year\": {\n", + " \"type\": \"number\",\n", + " \"description\": \"Draft year, e.g., 2025\"\n", + " },\n", + " \"includeAgentInfo\": {\n", + " \"type\": \"boolean\",\n", + " \"description\": \"Include agent information\"\n", + " },\n", + " \"includeStats\": {\n", + " \"type\": \"boolean\",\n", + " \"description\": \"Include detailed player statistics\"\n", + " },\n", + " \"includeScoutingReport\": {\n", + " \"type\": \"boolean\",\n", + " \"description\": \"Include scouting report details\"\n", + " }\n", + " },\n", + " \"required\": [\"playerName\", \"year\"]\n", + " }\n", + " }\n", + " ],\n", + " \"tool_choice\": \"auto\"\n", "}\n", - "\n", - "# Additional harness setup:\n", - "# - Add your repo to /testbed\n", - "# - Add your issue to the first user message\n", - "# - Note: Even though we used a single tool for python, bash, and apply_patch, we generally recommend defining more granular tools that are focused on a single function\n", - "\n", - "response = client.responses.create(\n", - " instructions=SYS_PROMPT_SWEBENCH,\n", - " model=\"gpt-4.1-2025-04-14\",\n", - " tools=[python_bash_patch_tool],\n", - " input=f\"Please answer the following question:\\nBug: Typerror...\"\n", - ")\n", - "\n", - "response.to_dict()[\"output\"]" + "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "# 2. Long context\n", + "### 2. As conversations unfold, optimize the context\n", "\n", - "GPT-4.1 has a performant 1M token input context window, and is useful for a variety of long context tasks, including structured document parsing, re-ranking, selecting relevant information while ignoring irrelevant context, and performing multi-hop reasoning using context.\n" + "While Realtime conversations support generous 30-minute sessions and a 128k token context window, you may notice performance gradually declining during extended exchanges. As conversations progress and more function calls are made, the conversation state can expand quickly with both important information and unnecessary noise—so it’s important to focus on keeping the most relevant details. This approach helps maintain strong performance and reduces cost." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Optimal Context Size\n", - "\n", - "We observe very good performance on needle-in-a-haystack evaluations up to our full 1M token context, and we’ve observed very strong performance at complex tasks with a mix of both relevant and irrelevant code and other documents. However, long context performance can degrade as more items are required to be retrieved, or perform complex reasoning that requires knowledge of the state of the entire context (like performing a graph search, for example).\n", - "\n", - "## Tuning Context Reliance\n", + "**i) Periodically summarize the conversation state**\n", "\n", - "Consider the mix of external vs. internal world knowledge that might be required to answer your question. Sometimes it’s important for the model to use some of its own knowledge to connect concepts or make logical jumps, while in others it’s desirable to only use provided context\n", + "Periodically summarizing the conversation as it unfolds is an excellent way to reduce context size—cutting both cost and latency.\n", "\n", - "```\n", - "# Instructions\n", - "// for internal knowledge\n", - "- Only use the documents in the provided External Context to answer the User Query. If you don't know the answer based on this context, you must respond \"I don't have the information needed to answer that\", even if a user insists on you answering the question.\n", - "// For internal and external knowledge\n", - "- By default, use the provided external context to answer the User Query, but if other basic knowledge is needed to answer, and you're confident in the answer, you can use some of your own knowledge to help answer the question.\n", - "```\n", - "\n", - "## Prompt Organization\n", - "\n", - "Especially in long context usage, placement of instructions and context can impact performance. If you have long context in your prompt, ideally place your instructions at both the beginning and end of the provided context, as we found this to perform better than only above or below. If you’d prefer to only have your instructions once, then above the provided context works better than below." + "See @Minhajul's' epic guide on implementing automatic summarization in Realtime conversations ([link](https://cookbook.openai.com/examples/context_summarization_with_realtime_api))." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "# 3. Chain of Thought\n", - "\n", - "As mentioned above, GPT-4.1 is not a reasoning model, but prompting the model to think step by step (called “chain of thought”) can be an effective way for a model to break down problems into more manageable pieces, solve them, and improve overall output quality, with the tradeoff of higher cost and latency associated with using more output tokens. The model has been trained to perform well at agentic reasoning about and real-world problem solving, so it shouldn’t require much prompting to perform well.\n", - "\n", - "We recommend starting with this basic chain-of-thought instruction at the end of your prompt:\n", - "\n", - "```\n", - "...\n", - "\n", - "First, think carefully step by step about what documents are needed to answer the query. Then, print out the TITLE and ID of each document. Then, format the IDs into a list.\n", - "```\n", + "**ii) Periodically remind the the model of its role and responsibilities**\n", "\n", - "From there, you should improve your chain-of-thought (CoT) prompt by auditing failures in your particular examples and evals, and addressing systematic planning and reasoning errors with more explicit instructions. In the unconstrained CoT prompt, there may be variance in the strategies it tries, and if you observe an approach that works well, you can codify that strategy in your prompt. Generally speaking, errors tend to occur from misunderstanding user intent, insufficient context gathering or analysis, or insufficient or incorrect step by step thinking, so watch out for these and try to address them with more opinionated instructions.\n", - "\n", - "Here is an example prompt instructing the model to focus more methodically on analyzing user intent and considering relevant context before proceeding to answer.\n", - "\n", - "```\n", - "# Reasoning Strategy\n", - "1. Query Analysis: Break down and analyze the query until you're confident about what it might be asking. Consider the provided context to help clarify any ambiguous or confusing information.\n", - "2. Context Analysis: Carefully select and analyze a large set of potentially relevant documents. Optimize for recall - it's okay if some are irrelevant, but the correct documents must be in this list, otherwise your final answer will be wrong. Analysis steps for each:\n", - "\ta. Analysis: An analysis of how it may or may not be relevant to answering the query.\n", - "\tb. Relevance rating: [high, medium, low, none]\n", - "3. Synthesis: summarize which documents are most relevant and why, including all documents with a relevance rating of medium or higher.\n", - "\n", - "# User Question\n", - "{user_question}\n", - "\n", - "# External Context\n", - "{external_context}\n", - "\n", - "First, think carefully step by step about what documents are needed to answer the query, closely adhering to the provided Reasoning Strategy. Then, print out the TITLE and ID of each document. Then, format the IDs into a list.\n", - "```\n" + "Data-heavy payloads can quickly fill the context window. If you notice the model losing track of instructions or available tools, periodically remind it of its system prompt and tools by calling `session.update`—this keeps it focused on its role and responsibilities." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "# 4. Instruction Following\n", - "\n", - "GPT-4.1 exhibits outstanding instruction-following performance, which developers can leverage to precisely shape and control the outputs for their particular use cases. Developers often extensively prompt for agentic reasoning steps, response tone and voice, tool calling information, output formatting, topics to avoid, and more. However, since the model follows instructions more literally, developers may need to include explicit specification around what to do or not to do. Furthermore, existing prompts optimized for other models may not immediately work with this model, because existing instructions are followed more closely and implicit rules are no longer being as strongly inferred." + "### 3. Data processing and optimization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Recommended Workflow\n", - "\n", - "Here is our recommended workflow for developing and debugging instructions in prompts:\n", - "\n", - "1. Start with an overall “Response Rules” or “Instructions” section with high-level guidance and bullet points. \n", - "2. If you’d like to change a more specific behavior, add a section to specify more details for that category, like `# Sample Phrases`. \n", - "3. If there are specific steps you’d like the model to follow in its workflow, add an ordered list and instruct the model to follow these steps.\n", - "4. If behavior still isn’t working as expected: \n", - " 1. Check for conflicting, underspecified, or wrong instructions and examples. If there are conflicting instructions, GPT-4.1 tends to follow the one closer to the end of the prompt.\n", - " 2. Add examples that demonstrate desired behavior; ensure that any important behavior demonstrated in your examples are also cited in your rules.\n", - " 3. It’s generally not necessary to use all-caps or other incentives like bribes or tips. We recommend starting without these, and only reaching for these if necessary for your particular prompt. Note that if your existing prompts include these techniques, it could cause GPT-4.1 to pay attention to it too strictly.\n", - "\n", - "*Note that using your preferred AI-powered IDE can be very helpful for iterating on prompts, including checking for consistency or conflicts, adding examples, or making cohesive updates like adding an instruction and updating instructions to demonstrate that instruction.*\n", + "**i) Use filtering in your function calls to trim data-heavy responses down to only the essential fields needed to answer the question**\n", "\n", - "## Common Failure Modes\n", + "Generally, fewer tokens returned by function calls lead to better quality responses. Common pitfalls occur when function calls return excessively large payloads spanning thousands of tokens. Focus on applying filters in each function call, either at the data-level or function-level, to minimize response sizes.\n", "\n", - "These failure modes are not unique to GPT-4.1, but we share them here for general awareness and ease of debugging.\n", - "\n", - "* Instructing a model to always follow a specific behavior can occasionally induce adverse effects. For instance, if told “you must call a tool before responding to the user,” models may hallucinate tool inputs or call the tool with null values if they do not have enough information. Adding “if you don’t have enough information to call the tool, ask the user for the information you need” should mitigate this.\n", - "* When provided sample phrases, models can use those quotes verbatim and start to sound repetitive to users. Ensure you instruct the model to vary them as necessary.\n", - "* Without specific instructions, some models can be eager to provide additional prose to explain their decisions, or output more formatting in responses than may be desired. Provide instructions and potentially examples to help mitigate." + "```json\n", + "// Filtered response\n", + "{\n", + " \"status\": {\n", + " \"code\": 200,\n", + " \"message\": \"SUCCESS\"\n", + " },\n", + " \"found\": 4274,\n", + " \"offset\": 0,\n", + " \"limit\": 5,\n", + " \"data\": [\n", + " {\n", + " \"zpid\": 7972122,\n", + " \"data\": {\n", + " \"PropertyInfo\": {\n", + " \"houseNumber\": \"19661\",\n", + " \"directionPrefix\": \"N \",\n", + " \"streetName\": \"Central\",\n", + " \"streetSuffix\": \"Ave\",\n", + " \"city\": \"Phoenix\",\n", + " \"state\": \"AZ\",\n", + " \"postalCode\": \"85024\",\n", + " \"zipPlusFour\": \"1641\"\n", + " \"bedroomCount\": 2,\n", + " \"bathroomCount\": 2,\n", + " \"storyCount\": 1,\n", + " \"livingAreaSize\": 1089,\n", + " \"livingAreaSizeUnits\": \"Square Feet\",\n", + " \"yearBuilt\": \"1985\"\n", + " }\n", + "\t\t }\n", + "\t\t\t}\n", + "\t\t]\n", + "\t\t// ... \n", + "}\n", + "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Example Prompt: Customer Service\n", - "\n", - "This demonstrates best practices for a fictional customer service agent. Observe the diversity of rules, the specificity, the use of additional sections for greater detail, and an example to demonstrate precise behavior that incorporates all prior rules.\n", - "\n", - "Try running the following notebook cell - you should see both a user message and tool call, and the user message should start with a greeting, then echo back their answer, then mention they're about to call a tool. Try changing the instructions to shape the model behavior, or trying other user messages, to test instruction following performance." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[{'id': 'msg_67fe92d431548191b7ca6cd604b4784b06efc5beb16b3c5e',\n", - " 'content': [{'annotations': [],\n", - " 'text': \"Hi, you've reached NewTelco, how can I help you? 🌍✈️\\n\\nYou'd like to know the cost of international service while traveling to France. 🇫🇷 Let me check the latest details for you—one moment, please. 🕑\",\n", - " 'type': 'output_text'}],\n", - " 'role': 'assistant',\n", - " 'status': 'completed',\n", - " 'type': 'message'},\n", - " {'arguments': '{\"topic\":\"international service cost France\"}',\n", - " 'call_id': 'call_cF63DLeyhNhwfdyME3ZHd0yo',\n", - " 'name': 'lookup_policy_document',\n", - " 'type': 'function_call',\n", - " 'id': 'fc_67fe92d5d6888191b6cd7cf57f707e4606efc5beb16b3c5e',\n", - " 'status': 'completed'}]" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "SYS_PROMPT_CUSTOMER_SERVICE = \"\"\"You are a helpful customer service agent working for NewTelco, helping a user efficiently fulfill their request while adhering closely to provided guidelines.\n", - "\n", - "# Instructions\n", - "- Always greet the user with \"Hi, you've reached NewTelco, how can I help you?\"\n", - "- Always call a tool before answering factual questions about the company, its offerings or products, or a user's account. Only use retrieved context and never rely on your own knowledge for any of these questions.\n", - " - However, if you don't have enough information to properly call the tool, ask the user for the information you need.\n", - "- Escalate to a human if the user requests.\n", - "- Do not discuss prohibited topics (politics, religion, controversial current events, medical, legal, or financial advice, personal conversations, internal company operations, or criticism of any people or company).\n", - "- Rely on sample phrases whenever appropriate, but never repeat a sample phrase in the same conversation. Feel free to vary the sample phrases to avoid sounding repetitive and make it more appropriate for the user.\n", - "- Always follow the provided output format for new messages, including citations for any factual statements from retrieved policy documents.\n", - "- If you're going to call a tool, always message the user with an appropriate message before and after calling the tool.\n", - "- Maintain a professional and concise tone in all responses, and use emojis between sentences.\n", - "- If you've resolved the user's request, ask if there's anything else you can help with\n", - "\n", - "# Precise Response Steps (for each response)\n", - "1. If necessary, call tools to fulfill the user's desired action. Always message the user before and after calling a tool to keep them in the loop.\n", - "2. In your response to the user\n", - " a. Use active listening and echo back what you heard the user ask for.\n", - " b. Respond appropriately given the above guidelines.\n", - "\n", - "# Sample Phrases\n", - "## Deflecting a Prohibited Topic\n", - "- \"I'm sorry, but I'm unable to discuss that topic. Is there something else I can help you with?\"\n", - "- \"That's not something I'm able to provide information on, but I'm happy to help with any other questions you may have.\"\n", - "\n", - "## Before calling a tool\n", - "- \"To help you with that, I'll just need to verify your information.\"\n", - "- \"Let me check that for you—one moment, please.\"\n", - "- \"I'll retrieve the latest details for you now.\"\n", - "\n", - "## After calling a tool\n", - "- \"Okay, here's what I found: [response]\"\n", - "- \"So here's what I found: [response]\"\n", - "\n", - "# Output Format\n", - "- Always include your final response to the user.\n", - "- When providing factual information from retrieved context, always include citations immediately after the relevant statement(s). Use the following citation format:\n", - " - For a single source: [NAME](ID)\n", - " - For multiple sources: [NAME](ID), [NAME](ID)\n", - "- Only provide information about this company, its policies, its products, or the customer's account, and only if it is based on information provided in context. Do not answer questions outside this scope.\n", + "**ii) Flatten hierarchical payloads—without losing key information**\n", "\n", - "# Example\n", - "## User\n", - "Can you tell me about your family plan options?\n", + "Hierarchical payloads from API calls can sometimes include repeated level titles—like \"ProspectInfo\" or \"Stats\"—which may add extra noise and make things harder for the model to process. As you explore ways to make your data more efficient, you might try flattening these structures by trimming away some of the unnecessary labels. This can help improve performance, but consider what information is important to keep for your particular use case.\n", "\n", - "## Assistant Response 1\n", - "### Message\n", - "\"Hi, you've reached NewTelco, how can I help you? 😊🎉\\n\\nYou'd like to know about our family plan options. 🤝 Let me check that for you—one moment, please. 🚀\"\n", - "\n", - "### Tool Calls\n", - "lookup_policy_document(topic=\"family plan options\")\n", - "\n", - "// After tool call, the assistant would follow up with:\n", - "\n", - "## Assistant Response 2 (after tool call)\n", - "### Message\n", - "\"Okay, here's what I found: 🎉 Our family plan allows up to 5 lines with shared data and a 10% discount for each additional line [Family Plan Policy](ID-010). 📱 Is there anything else I can help you with today? 😊\"\n", - "\"\"\"\n", - "\n", - "get_policy_doc = {\n", - " \"type\": \"function\",\n", - " \"name\": \"lookup_policy_document\",\n", - " \"description\": \"Tool to look up internal documents and policies by topic or keyword.\",\n", - " \"parameters\": {\n", - " \"strict\": True,\n", - " \"type\": \"object\",\n", - " \"properties\": {\n", - " \"topic\": {\n", - " \"type\": \"string\",\n", - " \"description\": \"The topic or keyword to search for in company policies or documents.\",\n", - " },\n", - " },\n", - " \"required\": [\"topic\"],\n", - " \"additionalProperties\": False,\n", - " },\n", - "}\n", - "\n", - "get_user_acct = {\n", - " \"type\": \"function\",\n", - " \"name\": \"get_user_account_info\",\n", - " \"description\": \"Tool to get user account information\",\n", - " \"parameters\": {\n", - " \"strict\": True,\n", - " \"type\": \"object\",\n", - " \"properties\": {\n", - " \"phone_number\": {\n", - " \"type\": \"string\",\n", - " \"description\": \"Formatted as '(xxx) xxx-xxxx'\",\n", - " },\n", - " },\n", - " \"required\": [\"phone_number\"],\n", - " \"additionalProperties\": False,\n", + "```json\n", + "// Flattened payload\n", + "{\n", + " \"status\": {\n", + " \"code\": 200,\n", + " \"message\": \"SUCCESS\"\n", + " },\n", + " \"found\": 4274,\n", + " \"offset\": 0,\n", + " \"limit\": 2,\n", + " \"data\": [\n", + " {\n", + " \"prospectId\": 10001,\n", + " \"league\": \"NCAA\",\n", + " \"collegeId\": 301,\n", + " \"isDraftEligible\": true,\n", + " \"firstName\": \"Jalen\",\n", + " \"lastName\": \"Storm\",\n", + " \"position\": \"PG\",\n", + " \"heightFeet\": 6,\n", + " \"heightInches\": 4,\n", + " \"weightPounds\": 205,\n", + " \"hometown\": \"Springfield\",\n", + " \"state\": \"IL\",\n", + " \"collegeTeam\": \"Springfield Tigers\",\n", + " \"conference\": \"Big West\",\n", + " \"teamRanking\": 12,\n", + " \"coachId\": 987,\n", + " \"coachName\": \"Marcus Reed\",\n", + " \"gamesPlayed\": 32,\n", + " \"minutesPerGame\": 34.5,\n", + " \"FieldGoalPercentage\": 47.2,\n", + " \"ThreePointPercentage\": 39.1,\n", + " \"FreeThrowPercentage\": 85.6,\n", + " \"averagePoints\": 21.3,\n", + " \"averageRebounds\": 4.1,\n", + " \"averageAssists\": 6.8,\n", + " \"stealsPerGame\": 1.7,\n", + " \"blocksPerGame\": 0.3,\n", + " \"strengths\": [\"Court vision\", \"Clutch shooting\"],\n", + " \"areasForImprovement\": [\"Defensive consistency\"],\n", + " \"mockDraftRanking\": 5,\n", + " \"lotteryPickProbability\": 88,\n", + " \"highlightReelUrl\": \"https://example.com/highlights/jalen-storm\",\n", + " \"agentName\": \"Rick Allen\",\n", + " \"agency\": \"Elite Sports Management\",\n", + " \"contactEmail\": \"rallen@elitesports.com\"\n", " },\n", - "}\n", - "\n", - "response = client.responses.create(\n", - " instructions=SYS_PROMPT_CUSTOMER_SERVICE,\n", - " model=\"gpt-4.1-2025-04-14\",\n", - " tools=[get_policy_doc, get_user_acct],\n", - " input=\"How much will it cost for international service? I'm traveling to France.\",\n", - " # input=\"Why was my last bill so high?\"\n", - ")\n", - "\n", - "response.to_dict()[\"output\"]" + "\t\t...\n", + " }\n", + " ```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "# 5. General Advice\n", - "\n", - "## Prompt Structure\n", - "\n", - "For reference, here is a good starting point for structuring your prompts.\n", - "\n", - "```\n", - "# Role and Objective\n", - "\n", - "# Instructions\n", - "\n", - "## Sub-categories for more detailed instructions\n", - "\n", - "# Reasoning Steps\n", - "\n", - "# Output Format\n", - "\n", - "# Examples\n", - "## Example 1\n", - "\n", - "# Context\n", - "\n", - "# Final instructions and prompt to think step by step\n", - "```\n", - "\n", - "Add or remove sections to suit your needs, and experiment to determine what’s optimal for your usage.\n", - "\n", - "## Delimiters\n", - "\n", - "Here are some general guidelines for selecting the best delimiters for your prompt. Please refer to the Long Context section for special considerations for that context type.\n", - "\n", - "1. Markdown: We recommend starting here, and using markdown titles for major sections and subsections (including deeper hierarchy, to H4+). Use inline backticks or backtick blocks to precisely wrap code, and standard numbered or bulleted lists as needed. \n", - "2. XML: These also perform well, and we have improved adherence to information in XML with this model. XML is convenient to precisely wrap a section including start and end, add metadata to the tags for additional context, and enable nesting. Here is an example of using XML tags to nest examples in an example section, with inputs and outputs for each:\n", - "\n", - "```\n", - "\n", - "\n", - "San Francisco\n", - "- SF\n", - "\n", - "\n", - "```\n", - "\n", - "3. JSON is highly structured and well understood by the model particularly in coding contexts. However it can be more verbose, and require character escaping that can add overhead.\n", - "\n", - "Guidance specifically for adding a large number of documents or files to input context:\n", - "\n", - "* XML performed well in our long context testing. \n", - " * Example: `The quick brown fox jumps over the lazy dog` \n", - "* This format, proposed by Lee et al. ([ref](https://arxiv.org/pdf/2406.13121)), also performed well in our long context testing. \n", - " * Example: `ID: 1 | TITLE: The Fox | CONTENT: The quick brown fox jumps over the lazy dog` \n", - "* JSON performed particularly poorly. \n", - " * Example: `[{'id': 1, 'title': 'The Fox', 'content': 'The quick brown fox jumped over the lazy dog'}]`\n", - "\n", - "The model is trained to robustly understand structure in a variety of formats. Generally, use your judgement and think about what will provide clear information and “stand out” to the model. For example, if you’re retrieving documents that contain lots of XML, an XML-based delimiter will likely be less effective. \n", - "\n", - "## Caveats\n", - "\n", - "* In some isolated cases we have observed the model being resistant to producing very long, repetitive outputs, for example, analyzing hundreds of items one by one. If this is necessary for your use case, instruct the model strongly to output this information in full, and consider breaking down the problem or using a more concise approach. \n", - "* We have seen some rare instances of parallel tool calls being incorrect. We advise testing this, and considering setting the [parallel\\_tool\\_calls](https://platform.openai.com/docs/api-reference/responses/create#responses-create-parallel_tool_calls) param to false if you’re seeing issues." + "**iii) Experiment with different data formats**\n", + "\n", + "The way you structure your data has a direct impact on how well the model processes and summarizes API responses. In our experience, clear, key-based formats like JSON or YAML help the model interpret data more accurately than tabular formats such as Markdown. Large tables, especially, tend to overwhelm the model—resulting in less fluent and less accurate outputs. Still, it’s worth experimenting with different formats to find what works best for your use case.\n", + "\n", + "```yaml\n", + "status:\n", + " code: 200\n", + " message: \"SUCCESS\"\n", + "found: 4274\n", + "offset: 0\n", + "limit: 10\n", + "data:\n", + " - prospectId: 10001\n", + " data:\n", + " ProspectInfo:\n", + " league: \"NCAA\"\n", + " collegeId: 301\n", + " isDraftEligible: true\n", + " Player:\n", + " firstName: \"Jalen\"\n", + " lastName: \"Storm\"\n", + " position: \"PG\"\n", + " heightFeet: 6\n", + " heightInches: 4\n", + " weightPounds: 205\n", + " hometown: \"Springfield\"\n", + " state: \"IL\"\n", + " TeamInfo:\n", + " collegeTeam: \"Springfield Tigers\"\n", + " conference: \"Big West\"\n", + " teamRanking: 12\n", + " coachId: 987\n", + " coachName: \"Marcus Reed\"\n", + " Stats:\n", + " gamesPlayed: 32\n", + " minutesPerGame: 34.5\n", + " FieldGoalPercentage: 47.2\n", + " ThreePointPercentage: 39.1\n", + " FreeThrowPercentage: 85.6\n", + " averagePoints: 21.3\n", + " averageRebounds: 4.1\n", + " averageAssists: 6.8\n", + " stealsPerGame: 1.7\n", + " blocksPerGame: 0.3\n", + " Scouting:\n", + " strengths:\n", + " - \"Court vision\"\n", + " - \"Clutch shooting\"\n", + " areasForImprovement:\n", + " - \"Defensive consistency\"\n", + " DraftProjection:\n", + " mockDraftRanking: 5\n", + " lotteryPickProbability: 88\n", + " Media:\n", + " highlightReelUrl: \"https://example.com/highlights/jalen-storm\"\n", + " Agent:\n", + " agentName: \"Rick Allen\"\n", + " agency: \"Elite Sports Management\"\n", + " contactEmail: \"rallen@elitesports.com\"\n", + "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "# Appendix: Generating and Applying File Diffs\n", - "\n", - "Developers have provided us feedback that accurate and well-formed diff generation is a critical capability to power coding-related tasks. To this end, the GPT-4.1 family features substantially improved diff capabilities relative to previous GPT models. Moreover, while GPT-4.1 has strong performance generating diffs of any format given clear instructions and examples, we open-source here one recommended diff format, on which the model has been extensively trained. We hope that in particular for developers just starting out, that this will take much of the guesswork out of creating diffs yourself. \n", - "\n", - "## Apply Patch\n", - "\n", - "See the example below for a prompt that applies our recommended tool call correctly." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "APPLY_PATCH_TOOL_DESC = \"\"\"This is a custom utility that makes it more convenient to add, remove, move, or edit code files. `apply_patch` effectively allows you to execute a diff/patch against a file, but the format of the diff specification is unique to this task, so pay careful attention to these instructions. To use the `apply_patch` command, you should pass a message of the following structure as \"input\":\n", - "\n", - "%%bash\n", - "apply_patch <<\"EOF\"\n", - "*** Begin Patch\n", - "[YOUR_PATCH]\n", - "*** End Patch\n", - "EOF\n", - "\n", - "Where [YOUR_PATCH] is the actual content of your patch, specified in the following V4A diff format.\n", - "\n", - "*** [ACTION] File: [path/to/file] -> ACTION can be one of Add, Update, or Delete.\n", - "For each snippet of code that needs to be changed, repeat the following:\n", - "[context_before] -> See below for further instructions on context.\n", - "- [old_code] -> Precede the old code with a minus sign.\n", - "+ [new_code] -> Precede the new, replacement code with a plus sign.\n", - "[context_after] -> See below for further instructions on context.\n", - "\n", - "For instructions on [context_before] and [context_after]:\n", - "- By default, show 3 lines of code immediately above and 3 lines immediately below each change. If a change is within 3 lines of a previous change, do NOT duplicate the first change’s [context_after] lines in the second change’s [context_before] lines.\n", - "- If 3 lines of context is insufficient to uniquely identify the snippet of code within the file, use the @@ operator to indicate the class or function to which the snippet belongs. For instance, we might have:\n", - "@@ class BaseClass\n", - "[3 lines of pre-context]\n", - "- [old_code]\n", - "+ [new_code]\n", - "[3 lines of post-context]\n", - "\n", - "- If a code block is repeated so many times in a class or function such that even a single @@ statement and 3 lines of context cannot uniquely identify the snippet of code, you can use multiple `@@` statements to jump to the right context. For instance:\n", - "\n", - "@@ class BaseClass\n", - "@@ \tdef method():\n", - "[3 lines of pre-context]\n", - "- [old_code]\n", - "+ [new_code]\n", - "[3 lines of post-context]\n", - "\n", - "Note, then, that we do not use line numbers in this diff format, as the context is enough to uniquely identify code. An example of a message that you might pass as \"input\" to this function, in order to apply a patch, is shown below.\n", - "\n", - "%%bash\n", - "apply_patch <<\"EOF\"\n", - "*** Begin Patch\n", - "*** Update File: pygorithm/searching/binary_search.py\n", - "@@ class BaseClass\n", - "@@ def search():\n", - "- pass\n", - "+ raise NotImplementedError()\n", - "\n", - "@@ class Subclass\n", - "@@ def search():\n", - "- pass\n", - "+ raise NotImplementedError()\n", - "\n", - "*** End Patch\n", - "EOF\n", - "\"\"\"\n", - "\n", - "APPLY_PATCH_TOOL = {\n", - " \"name\": \"apply_patch\",\n", - " \"description\": APPLY_PATCH_TOOL_DESC,\n", - " \"parameters\": {\n", - " \"type\": \"object\",\n", - " \"properties\": {\n", - " \"input\": {\n", - " \"type\": \"string\",\n", - " \"description\": \" The apply_patch command that you wish to execute.\",\n", - " }\n", - " },\n", - " \"required\": [\"input\"],\n", - " },\n", - "}" + "## 4. After data-heavy function calls, follow up with hint prompts\n", + "\n", + "Underlying models often struggle to transition smoothly from data-heavy responses to accurate answers. To improve fluency and accuracy when working with complex data, provide a function call hint immediately after the function call. These hints guide the model on the specific task—teaching it how to interpret key fields and domain-specific values.\n", + "\n", + "The following example illustrates an effective hint prompt.\n", + "\n", + "```javascript\n", + "// Function call hint\n", + "let prospectSearchPrompt = `\n", + "Parse NBA prospect data and provide a concise, engaging response.\n", + "\n", + "General Guidelines\n", + "- Act as an NBA scouting expert.\n", + "- Highlight key strengths and notable attributes.\n", + "- Use conversational language.\n", + "- Mention identical attributes once.\n", + "- Ignore IDs and URLs.\n", + "\n", + "Player Details\n", + "- State height conversationally (\"six-foot-eight\").\n", + "- Round weights to nearest 5 lbs.\n", + "\n", + "Stats & Draft Info\n", + "- Round stats to nearest whole number.\n", + "- Use general terms for draft ranking (\"top-five pick\").\n", + "Experience\n", + "- Refer to players as freshman, sophomore, etc., or mention professional experience.\n", + "- Location & TeamMention hometown city and state/country.\n", + "- Describe teams conversationally.\n", + "\n", + "Skip (unless asked explicitly)\n", + "- Exact birth dates\n", + "- IDs\n", + "- Agent/contact details\n", + "- URLs\n", + "\n", + "Examples\n", + "- \"Jalen Storm, a dynamic six-foot-four point guard from Springfield, Illinois, averages 21 points per game.\"\n", + "- \"Known for his clutch shooting, he's projected as a top-five pick.\"\n", + "\n", + "Important: Respond based strictly on provided data, without inventing details.\n", + "`;\n", + "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Reference Implementation: apply\\_patch.py\n", - "\n", - "Here’s a reference implementation of the apply\\_patch tool that we used as part of model training. You’ll need to make this an executable and available as \\`apply\\_patch\\` from the shell where the model will execute commands:" + "In practice, we first append the function call result to the conversation. Then, we emit a response from the Realtime API with the hint prompt. Voilà—the model gracefully handles all the information." ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ - "#!/usr/bin/env python3\n", - "\n", - "\"\"\"\n", - "A self-contained **pure-Python 3.9+** utility for applying human-readable\n", - "“pseudo-diff” patch files to a collection of text files.\n", - "\"\"\"\n", - "\n", - "from __future__ import annotations\n", - "\n", - "import pathlib\n", - "from dataclasses import dataclass, field\n", - "from enum import Enum\n", - "from typing import (\n", - " Callable,\n", - " Dict,\n", - " List,\n", - " Optional,\n", - " Tuple,\n", - " Union,\n", - ")\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# Domain objects\n", - "# --------------------------------------------------------------------------- #\n", - "class ActionType(str, Enum):\n", - " ADD = \"add\"\n", - " DELETE = \"delete\"\n", - " UPDATE = \"update\"\n", - "\n", - "\n", - "@dataclass\n", - "class FileChange:\n", - " type: ActionType\n", - " old_content: Optional[str] = None\n", - " new_content: Optional[str] = None\n", - " move_path: Optional[str] = None\n", - "\n", - "\n", - "@dataclass\n", - "class Commit:\n", - " changes: Dict[str, FileChange] = field(default_factory=dict)\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# Exceptions\n", - "# --------------------------------------------------------------------------- #\n", - "class DiffError(ValueError):\n", - " \"\"\"Any problem detected while parsing or applying a patch.\"\"\"\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# Helper dataclasses used while parsing patches\n", - "# --------------------------------------------------------------------------- #\n", - "@dataclass\n", - "class Chunk:\n", - " orig_index: int = -1\n", - " del_lines: List[str] = field(default_factory=list)\n", - " ins_lines: List[str] = field(default_factory=list)\n", - "\n", - "\n", - "@dataclass\n", - "class PatchAction:\n", - " type: ActionType\n", - " new_file: Optional[str] = None\n", - " chunks: List[Chunk] = field(default_factory=list)\n", - " move_path: Optional[str] = None\n", - "\n", - "\n", - "@dataclass\n", - "class Patch:\n", - " actions: Dict[str, PatchAction] = field(default_factory=dict)\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# Patch text parser\n", - "# --------------------------------------------------------------------------- #\n", - "@dataclass\n", - "class Parser:\n", - " current_files: Dict[str, str]\n", - " lines: List[str]\n", - " index: int = 0\n", - " patch: Patch = field(default_factory=Patch)\n", - " fuzz: int = 0\n", - "\n", - " # ------------- low-level helpers -------------------------------------- #\n", - " def _cur_line(self) -> str:\n", - " if self.index >= len(self.lines):\n", - " raise DiffError(\"Unexpected end of input while parsing patch\")\n", - " return self.lines[self.index]\n", - "\n", - " @staticmethod\n", - " def _norm(line: str) -> str:\n", - " \"\"\"Strip CR so comparisons work for both LF and CRLF input.\"\"\"\n", - " return line.rstrip(\"\\r\")\n", - "\n", - " # ------------- scanning convenience ----------------------------------- #\n", - " def is_done(self, prefixes: Optional[Tuple[str, ...]] = None) -> bool:\n", - " if self.index >= len(self.lines):\n", - " return True\n", - " if (\n", - " prefixes\n", - " and len(prefixes) > 0\n", - " and self._norm(self._cur_line()).startswith(prefixes)\n", - " ):\n", - " return True\n", - " return False\n", - "\n", - " def startswith(self, prefix: Union[str, Tuple[str, ...]]) -> bool:\n", - " return self._norm(self._cur_line()).startswith(prefix)\n", - "\n", - " def read_str(self, prefix: str) -> str:\n", - " \"\"\"\n", - " Consume the current line if it starts with *prefix* and return the text\n", - " **after** the prefix. Raises if prefix is empty.\n", - " \"\"\"\n", - " if prefix == \"\":\n", - " raise ValueError(\"read_str() requires a non-empty prefix\")\n", - " if self._norm(self._cur_line()).startswith(prefix):\n", - " text = self._cur_line()[len(prefix) :]\n", - " self.index += 1\n", - " return text\n", - " return \"\"\n", - "\n", - " def read_line(self) -> str:\n", - " \"\"\"Return the current raw line and advance.\"\"\"\n", - " line = self._cur_line()\n", - " self.index += 1\n", - " return line\n", - "\n", - " # ------------- public entry point -------------------------------------- #\n", - " def parse(self) -> None:\n", - " while not self.is_done((\"*** End Patch\",)):\n", - " # ---------- UPDATE ---------- #\n", - " path = self.read_str(\"*** Update File: \")\n", - " if path:\n", - " if path in self.patch.actions:\n", - " raise DiffError(f\"Duplicate update for file: {path}\")\n", - " move_to = self.read_str(\"*** Move to: \")\n", - " if path not in self.current_files:\n", - " raise DiffError(f\"Update File Error - missing file: {path}\")\n", - " text = self.current_files[path]\n", - " action = self._parse_update_file(text)\n", - " action.move_path = move_to or None\n", - " self.patch.actions[path] = action\n", - " continue\n", - "\n", - " # ---------- DELETE ---------- #\n", - " path = self.read_str(\"*** Delete File: \")\n", - " if path:\n", - " if path in self.patch.actions:\n", - " raise DiffError(f\"Duplicate delete for file: {path}\")\n", - " if path not in self.current_files:\n", - " raise DiffError(f\"Delete File Error - missing file: {path}\")\n", - " self.patch.actions[path] = PatchAction(type=ActionType.DELETE)\n", - " continue\n", - "\n", - " # ---------- ADD ---------- #\n", - " path = self.read_str(\"*** Add File: \")\n", - " if path:\n", - " if path in self.patch.actions:\n", - " raise DiffError(f\"Duplicate add for file: {path}\")\n", - " if path in self.current_files:\n", - " raise DiffError(f\"Add File Error - file already exists: {path}\")\n", - " self.patch.actions[path] = self._parse_add_file()\n", - " continue\n", - "\n", - " raise DiffError(f\"Unknown line while parsing: {self._cur_line()}\")\n", - "\n", - " if not self.startswith(\"*** End Patch\"):\n", - " raise DiffError(\"Missing *** End Patch sentinel\")\n", - " self.index += 1 # consume sentinel\n", - "\n", - " # ------------- section parsers ---------------------------------------- #\n", - " def _parse_update_file(self, text: str) -> PatchAction:\n", - " action = PatchAction(type=ActionType.UPDATE)\n", - " lines = text.split(\"\\n\")\n", - " index = 0\n", - " while not self.is_done(\n", - " (\n", - " \"*** End Patch\",\n", - " \"*** Update File:\",\n", - " \"*** Delete File:\",\n", - " \"*** Add File:\",\n", - " \"*** End of File\",\n", - " )\n", - " ):\n", - " def_str = self.read_str(\"@@ \")\n", - " section_str = \"\"\n", - " if not def_str and self._norm(self._cur_line()) == \"@@\":\n", - " section_str = self.read_line()\n", - "\n", - " if not (def_str or section_str or index == 0):\n", - " raise DiffError(f\"Invalid line in update section:\\n{self._cur_line()}\")\n", - "\n", - " if def_str.strip():\n", - " found = False\n", - " if def_str not in lines[:index]:\n", - " for i, s in enumerate(lines[index:], index):\n", - " if s == def_str:\n", - " index = i + 1\n", - " found = True\n", - " break\n", - " if not found and def_str.strip() not in [\n", - " s.strip() for s in lines[:index]\n", - " ]:\n", - " for i, s in enumerate(lines[index:], index):\n", - " if s.strip() == def_str.strip():\n", - " index = i + 1\n", - " self.fuzz += 1\n", - " found = True\n", - " break\n", - "\n", - " next_ctx, chunks, end_idx, eof = peek_next_section(self.lines, self.index)\n", - " new_index, fuzz = find_context(lines, next_ctx, index, eof)\n", - " if new_index == -1:\n", - " ctx_txt = \"\\n\".join(next_ctx)\n", - " raise DiffError(\n", - " f\"Invalid {'EOF ' if eof else ''}context at {index}:\\n{ctx_txt}\"\n", - " )\n", - " self.fuzz += fuzz\n", - " for ch in chunks:\n", - " ch.orig_index += new_index\n", - " action.chunks.append(ch)\n", - " index = new_index + len(next_ctx)\n", - " self.index = end_idx\n", - " return action\n", - "\n", - " def _parse_add_file(self) -> PatchAction:\n", - " lines: List[str] = []\n", - " while not self.is_done(\n", - " (\"*** End Patch\", \"*** Update File:\", \"*** Delete File:\", \"*** Add File:\")\n", - " ):\n", - " s = self.read_line()\n", - " if not s.startswith(\"+\"):\n", - " raise DiffError(f\"Invalid Add File line (missing '+'): {s}\")\n", - " lines.append(s[1:]) # strip leading '+'\n", - " return PatchAction(type=ActionType.ADD, new_file=\"\\n\".join(lines))\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# Helper functions\n", - "# --------------------------------------------------------------------------- #\n", - "def find_context_core(\n", - " lines: List[str], context: List[str], start: int\n", - ") -> Tuple[int, int]:\n", - " if not context:\n", - " return start, 0\n", - "\n", - " for i in range(start, len(lines)):\n", - " if lines[i : i + len(context)] == context:\n", - " return i, 0\n", - " for i in range(start, len(lines)):\n", - " if [s.rstrip() for s in lines[i : i + len(context)]] == [\n", - " s.rstrip() for s in context\n", - " ]:\n", - " return i, 1\n", - " for i in range(start, len(lines)):\n", - " if [s.strip() for s in lines[i : i + len(context)]] == [\n", - " s.strip() for s in context\n", - " ]:\n", - " return i, 100\n", - " return -1, 0\n", - "\n", - "\n", - "def find_context(\n", - " lines: List[str], context: List[str], start: int, eof: bool\n", - ") -> Tuple[int, int]:\n", - " if eof:\n", - " new_index, fuzz = find_context_core(lines, context, len(lines) - len(context))\n", - " if new_index != -1:\n", - " return new_index, fuzz\n", - " new_index, fuzz = find_context_core(lines, context, start)\n", - " return new_index, fuzz + 10_000\n", - " return find_context_core(lines, context, start)\n", - "\n", - "\n", - "def peek_next_section(\n", - " lines: List[str], index: int\n", - ") -> Tuple[List[str], List[Chunk], int, bool]:\n", - " old: List[str] = []\n", - " del_lines: List[str] = []\n", - " ins_lines: List[str] = []\n", - " chunks: List[Chunk] = []\n", - " mode = \"keep\"\n", - " orig_index = index\n", - "\n", - " while index < len(lines):\n", - " s = lines[index]\n", - " if s.startswith(\n", - " (\n", - " \"@@\",\n", - " \"*** End Patch\",\n", - " \"*** Update File:\",\n", - " \"*** Delete File:\",\n", - " \"*** Add File:\",\n", - " \"*** End of File\",\n", - " )\n", - " ):\n", - " break\n", - " if s == \"***\":\n", - " break\n", - " if s.startswith(\"***\"):\n", - " raise DiffError(f\"Invalid Line: {s}\")\n", - " index += 1\n", - "\n", - " last_mode = mode\n", - " if s == \"\":\n", - " s = \" \"\n", - " if s[0] == \"+\":\n", - " mode = \"add\"\n", - " elif s[0] == \"-\":\n", - " mode = \"delete\"\n", - " elif s[0] == \" \":\n", - " mode = \"keep\"\n", - " else:\n", - " raise DiffError(f\"Invalid Line: {s}\")\n", - " s = s[1:]\n", - "\n", - " if mode == \"keep\" and last_mode != mode:\n", - " if ins_lines or del_lines:\n", - " chunks.append(\n", - " Chunk(\n", - " orig_index=len(old) - len(del_lines),\n", - " del_lines=del_lines,\n", - " ins_lines=ins_lines,\n", - " )\n", - " )\n", - " del_lines, ins_lines = [], []\n", - "\n", - " if mode == \"delete\":\n", - " del_lines.append(s)\n", - " old.append(s)\n", - " elif mode == \"add\":\n", - " ins_lines.append(s)\n", - " elif mode == \"keep\":\n", - " old.append(s)\n", - "\n", - " if ins_lines or del_lines:\n", - " chunks.append(\n", - " Chunk(\n", - " orig_index=len(old) - len(del_lines),\n", - " del_lines=del_lines,\n", - " ins_lines=ins_lines,\n", - " )\n", - " )\n", - "\n", - " if index < len(lines) and lines[index] == \"*** End of File\":\n", - " index += 1\n", - " return old, chunks, index, True\n", - "\n", - " if index == orig_index:\n", - " raise DiffError(\"Nothing in this section\")\n", - " return old, chunks, index, False\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# Patch → Commit and Commit application\n", - "# --------------------------------------------------------------------------- #\n", - "def _get_updated_file(text: str, action: PatchAction, path: str) -> str:\n", - " if action.type is not ActionType.UPDATE:\n", - " raise DiffError(\"_get_updated_file called with non-update action\")\n", - " orig_lines = text.split(\"\\n\")\n", - " dest_lines: List[str] = []\n", - " orig_index = 0\n", - "\n", - " for chunk in action.chunks:\n", - " if chunk.orig_index > len(orig_lines):\n", - " raise DiffError(\n", - " f\"{path}: chunk.orig_index {chunk.orig_index} exceeds file length\"\n", - " )\n", - " if orig_index > chunk.orig_index:\n", - " raise DiffError(\n", - " f\"{path}: overlapping chunks at {orig_index} > {chunk.orig_index}\"\n", - " )\n", - "\n", - " dest_lines.extend(orig_lines[orig_index : chunk.orig_index])\n", - " orig_index = chunk.orig_index\n", - "\n", - " dest_lines.extend(chunk.ins_lines)\n", - " orig_index += len(chunk.del_lines)\n", - "\n", - " dest_lines.extend(orig_lines[orig_index:])\n", - " return \"\\n\".join(dest_lines)\n", - "\n", - "\n", - "def patch_to_commit(patch: Patch, orig: Dict[str, str]) -> Commit:\n", - " commit = Commit()\n", - " for path, action in patch.actions.items():\n", - " if action.type is ActionType.DELETE:\n", - " commit.changes[path] = FileChange(\n", - " type=ActionType.DELETE, old_content=orig[path]\n", - " )\n", - " elif action.type is ActionType.ADD:\n", - " if action.new_file is None:\n", - " raise DiffError(\"ADD action without file content\")\n", - " commit.changes[path] = FileChange(\n", - " type=ActionType.ADD, new_content=action.new_file\n", - " )\n", - " elif action.type is ActionType.UPDATE:\n", - " new_content = _get_updated_file(orig[path], action, path)\n", - " commit.changes[path] = FileChange(\n", - " type=ActionType.UPDATE,\n", - " old_content=orig[path],\n", - " new_content=new_content,\n", - " move_path=action.move_path,\n", - " )\n", - " return commit\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# User-facing helpers\n", - "# --------------------------------------------------------------------------- #\n", - "def text_to_patch(text: str, orig: Dict[str, str]) -> Tuple[Patch, int]:\n", - " lines = text.splitlines() # preserves blank lines, no strip()\n", - " if (\n", - " len(lines) < 2\n", - " or not Parser._norm(lines[0]).startswith(\"*** Begin Patch\")\n", - " or Parser._norm(lines[-1]) != \"*** End Patch\"\n", - " ):\n", - " raise DiffError(\"Invalid patch text - missing sentinels\")\n", - "\n", - " parser = Parser(current_files=orig, lines=lines, index=1)\n", - " parser.parse()\n", - " return parser.patch, parser.fuzz\n", - "\n", - "\n", - "def identify_files_needed(text: str) -> List[str]:\n", - " lines = text.splitlines()\n", - " return [\n", - " line[len(\"*** Update File: \") :]\n", - " for line in lines\n", - " if line.startswith(\"*** Update File: \")\n", - " ] + [\n", - " line[len(\"*** Delete File: \") :]\n", - " for line in lines\n", - " if line.startswith(\"*** Delete File: \")\n", - " ]\n", - "\n", - "\n", - "def identify_files_added(text: str) -> List[str]:\n", - " lines = text.splitlines()\n", - " return [\n", - " line[len(\"*** Add File: \") :]\n", - " for line in lines\n", - " if line.startswith(\"*** Add File: \")\n", - " ]\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# File-system helpers\n", - "# --------------------------------------------------------------------------- #\n", - "def load_files(paths: List[str], open_fn: Callable[[str], str]) -> Dict[str, str]:\n", - " return {path: open_fn(path) for path in paths}\n", - "\n", - "\n", - "def apply_commit(\n", - " commit: Commit,\n", - " write_fn: Callable[[str, str], None],\n", - " remove_fn: Callable[[str], None],\n", - ") -> None:\n", - " for path, change in commit.changes.items():\n", - " if change.type is ActionType.DELETE:\n", - " remove_fn(path)\n", - " elif change.type is ActionType.ADD:\n", - " if change.new_content is None:\n", - " raise DiffError(f\"ADD change for {path} has no content\")\n", - " write_fn(path, change.new_content)\n", - " elif change.type is ActionType.UPDATE:\n", - " if change.new_content is None:\n", - " raise DiffError(f\"UPDATE change for {path} has no new content\")\n", - " target = change.move_path or path\n", - " write_fn(target, change.new_content)\n", - " if change.move_path:\n", - " remove_fn(path)\n", - "\n", - "\n", - "def process_patch(\n", - " text: str,\n", - " open_fn: Callable[[str], str],\n", - " write_fn: Callable[[str, str], None],\n", - " remove_fn: Callable[[str], None],\n", - ") -> str:\n", - " if not text.startswith(\"*** Begin Patch\"):\n", - " raise DiffError(\"Patch text must start with *** Begin Patch\")\n", - " paths = identify_files_needed(text)\n", - " orig = load_files(paths, open_fn)\n", - " patch, _fuzz = text_to_patch(text, orig)\n", - " commit = patch_to_commit(patch, orig)\n", - " apply_commit(commit, write_fn, remove_fn)\n", - " return \"Done!\"\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# Default FS helpers\n", - "# --------------------------------------------------------------------------- #\n", - "def open_file(path: str) -> str:\n", - " with open(path, \"rt\", encoding=\"utf-8\") as fh:\n", - " return fh.read()\n", - "\n", - "\n", - "def write_file(path: str, content: str) -> None:\n", - " target = pathlib.Path(path)\n", - " target.parent.mkdir(parents=True, exist_ok=True)\n", - " with target.open(\"wt\", encoding=\"utf-8\") as fh:\n", - " fh.write(content)\n", - "\n", - "\n", - "def remove_file(path: str) -> None:\n", - " pathlib.Path(path).unlink(missing_ok=True)\n", - "\n", - "\n", - "# --------------------------------------------------------------------------- #\n", - "# CLI entry-point\n", - "# --------------------------------------------------------------------------- #\n", - "def main() -> None:\n", - " import sys\n", + "```javascript\n", + "// Add new conversation item for the model\n", + "const conversationItem = {\n", + " type: 'conversation.item.create',\n", + " previous_item_id: output.id,\n", + " item: {\n", + " call_id: output.call_id,\n", + " type: 'function_call_output',\n", + " output: `Draft Prospect Search Results: ${result}`\n", + " }\n", + "};\n", "\n", - " patch_text = sys.stdin.read()\n", - " if not patch_text:\n", - " print(\"Please pass patch text through stdin\", file=sys.stderr)\n", - " return\n", - " try:\n", - " result = process_patch(patch_text, open_file, write_file, remove_file)\n", - " except DiffError as exc:\n", - " print(exc, file=sys.stderr)\n", - " return\n", - " print(result)\n", + "dataChannel.send(JSON.stringify(conversationItem));\n", "\n", + "// Emit a response from the model including the hint prompt\n", + "const event = {\n", + " type: 'response.create',\n", + " conversation: \"none\",\n", + " response: {\n", + " instructions: prospectSearchPrompt # function call hint\n", + " }\n", + "};\n", "\n", - "if __name__ == \"__main__\":\n", - " main()\n" + "dataChannel.send(JSON.stringify(event));\n", + "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Other Effective Diff Formats\n", + "## Wrapping up\n", "\n", - "If you want to try using a different diff format, we found in testing that the SEARCH/REPLACE diff format used in Aider’s polyglot benchmark, as well as a pseudo-XML format with no internal escaping, both had high success rates.\n", + "Building effective agents with the Realtime API is an ongoing process of exploration and adaptation.\n", "\n", - "These diff formats share two key aspects: (1) they do not use line numbers, and (2) they provide both the exact code to be replaced, and the exact code with which to replace it, with clear delimiters between the two." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "SEARCH_REPLACE_DIFF_EXAMPLE = \"\"\"\n", - "path/to/file.py\n", - "```\n", - ">>>>>>> SEARCH\n", - "def search():\n", - " pass\n", - "=======\n", - "def search():\n", - " raise NotImplementedError()\n", - "<<<<<<< REPLACE\n", - "\"\"\"\n", + "**Summary of Key Recommendations**\n", + "\n", + "- **Filter data:** Only include fields and details that are directly relevant to the user’s request or the model’s next step. Trim the rest.\n", + "- **Flatten and simplify structures:** Reduce deeply nested or redundant data. Present information in a way that’s easy for both models and humans to scan.\n", + "- **Prefer clear, structured formats:** Use JSON (or YAML) with consistent field names and minimal noise. Avoid large tables or markdown for data-heavy responses.\n", + "- **Guide the model with hint prompts:** After returning lots of data, follow up with a targeted prompt that explains exactly what the model should extract or summarize.\n", "\n", - "PSEUDO_XML_DIFF_EXAMPLE = \"\"\"\n", - "\n", - "\n", - "path/to/file.py\n", - "\n", - "\n", - "def search():\n", - " pass\n", - "\n", - "\n", - "def search():\n", - " raise NotImplementedError()\n", - "\n", - "\n", - "\"\"\"" + "Remember—experimentation is essential. Realtime models keep improving, and we’ll continue sharing tips to help you get the most out of the Realtime API." ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] } ], "metadata": { diff --git a/registry.yaml b/registry.yaml index 099ad1900e..8e135fdebb 100644 --- a/registry.yaml +++ b/registry.yaml @@ -2078,3 +2078,16 @@ - agents - agents-sdk - parallel-agents + +- title: Practical guide to data-intensive apps with the Realtime API + path: examples/Data-intensive-Realtime-apps.ipynb + date: 2025-05-29 + authors: + - alistair-openai + - rkoenig-openai + - phundal-openai + tags: + - realtime-api + - data-intensive + - agents-sdk + From f259ed1c31b7718c3b5c34383a372da8a21355db Mon Sep 17 00:00:00 2001 From: alistair-openai Date: Thu, 29 May 2025 11:14:13 -0400 Subject: [PATCH 3/3] update authors and registry --- authors.yaml | 10 +++++++++- examples/Data-intensive-Realtime-apps.ipynb | 2 +- registry.yaml | 5 ++--- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/authors.yaml b/authors.yaml index b44ff27ccd..634e0f5510 100644 --- a/authors.yaml +++ b/authors.yaml @@ -352,4 +352,12 @@ alistair-openai: website: "https://www.linkedin.com/in/alistair-gillespie/" avatar: "https://avatars.githubusercontent.com/u/210626148" - +phundal-openai: + name: "Patrick Hundal" + website: "https://www.linkedin.com/in/phundal/" + avatar: "https://avatars.githubusercontent.com/u/189161955" + +rkoenig-openai: + name: "Robin Koenig" + website: "https://www.linkedin.com/in/robinkoenig/" + avatar: "https://media.licdn.com/dms/image/v2/C5603AQEqUVtbts8Huw/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1558137883581?e=1753920000&v=beta&t=jcm-qNJfmgVJsS6uNHxHu5T2nQoUWkXivthzxTJMWqA" \ No newline at end of file diff --git a/examples/Data-intensive-Realtime-apps.ipynb b/examples/Data-intensive-Realtime-apps.ipynb index ed3753c0e5..a94390131e 100644 --- a/examples/Data-intensive-Realtime-apps.ipynb +++ b/examples/Data-intensive-Realtime-apps.ipynb @@ -291,7 +291,7 @@ "source": [ "### 2. As conversations unfold, optimize the context\n", "\n", - "While Realtime conversations support generous 30-minute sessions and a 128k token context window, you may notice performance gradually declining during extended exchanges. As conversations progress and more function calls are made, the conversation state can expand quickly with both important information and unnecessary noise—so it’s important to focus on keeping the most relevant details. This approach helps maintain strong performance and reduces cost." + "Realtime conversations allow for generous 30-minute sessions—but the rolling context window only supports ~16,000 tokens (depending on the model snapshot, context window limitations are improving). As a result, you may notice performance gradually decline during extended exchanges. As conversations progress and more function calls are made, the conversation state can expand quickly with both important information and unnecessary noise—so it’s important to focus on keeping the most relevant details. This approach helps maintain strong performance and reduces cost." ] }, { diff --git a/registry.yaml b/registry.yaml index 8e135fdebb..db612ac0e3 100644 --- a/registry.yaml +++ b/registry.yaml @@ -2087,7 +2087,6 @@ - rkoenig-openai - phundal-openai tags: - - realtime-api - - data-intensive - - agents-sdk + - audio + - speech