feat: include upstream cost forwarding #267
Conversation
Adds support for OpenRouter's BYOK usage accounting. If you have a provider's own API key in your OpenRouter account, cost details are now accessible via usage.costDetails.upstreamInferenceCost.
Enhances BYOK usage accounting with cost details access.
There was a problem hiding this comment.
Pull request overview
This PR adds support for OpenRouter's BYOK (Bring Your Own Key) upstream cost tracking to the provider's streaming and completion endpoints. When users configure their own provider API keys in OpenRouter (e.g., an OpenAI key), OpenRouter's usage accounting API returns upstream inference costs in cost_details.upstream_inference_cost. Previously, this field was only being forwarded in the chat doGenerate method (added in PR #107), but was missing from streaming endpoints and the completion API. This PR completes the implementation to ensure upstream costs are accessible across all API endpoints.
Key changes:
- Adds
cost_detailsschema support to completion streaming responses - Forwards
upstreamInferenceCostin chat and completion streamingfinishevents when present - Adds comprehensive test coverage for upstream cost scenarios (cost alone, upstream cost alone, and both together)
- Documents BYOK usage accounting feature in README with code examples
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/completion/schemas.ts | Adds cost_details schema with upstream_inference_cost field to support parsing BYOK cost data |
| src/completion/index.ts | Extracts and forwards upstream inference cost in streaming finish events when present |
| src/chat/index.ts | Extracts and forwards upstream inference cost in streaming finish events when present |
| src/completion/index.test.ts | Adds tests for upstream cost handling in completion streaming (cost alone and combined scenarios) |
| src/chat/index.test.ts | Adds tests for upstream cost handling in chat streaming (cost alone and combined scenarios) |
| src/tests/usage-accounting.test.ts | Fixes test data bug (19 → 0.0019) to use realistic cost values |
| src/tests/stream-usage-accounting.test.ts | Adds upstream cost to streaming test mock data |
| README.md | Documents BYOK usage accounting feature with code examples showing how to access upstream costs |
| .changeset/tricky-pumas-mix.md | Adds changeset describing the new BYOK usage accounting support |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Thank you @subtleGradient ! And thanks for the bonus tokendetails optional change 🙏 |
OpenRouter supports BYOK, where the user adds a provider API key to their OpenRouter account, and usage is then charged directly to that key. In that case, OpenRouter's usage accounting stores the upstream costs in
cost_details.upstream_inference_cost. This field was not being passed through this provider in all cases, and so users could not access these costs in their applications.This PR adds the upstream inference cost to the chat & completions outputs when it is present, and is backwards-compatible. It also adds tests for the same, including evaluating 1) only normal cost, 2) only upstream inference cost, and 3) both normal and upstream inference cost.
PR #107 added some of this behavior (for
doGenerate) but left it out of the streaming and completion endpoints. It also enforcedupstreamInferenceCost: 0even when OpenRouter never actually reported that field, which I'm not sure is the correct behavior (although I'm happy to revert that if it is). This PR also adds more robust testing.Re-created from #247 by @abromberg, rebased on main after #266 landed.
Co-authored-by: Andy Bromberg andy@andybromberg.com