fix(rivetkit): handle all top-level errors (#3578)

Author: NathanFlurry

.claude/agents/js-error-handler-auditor.md

@@ -0,0 +1,129 @@
+---
+name: js-error-handler-auditor
+description: Use this agent when you need to audit a JavaScript codebase for unhandled errors in top-level async operations, event handlers, and callback functions. This agent should be used proactively when:\n\n<example>\nContext: User wants to improve error handling across their JavaScript codebase.\nuser: "Can you check if we're handling errors properly in our event listeners?"\nassistant: "I'll use the Task tool to launch the js-error-handler-auditor agent to comprehensively audit error handling across your JavaScript codebase."\n<commentary>\nThe user is asking about error handling practices, which is exactly what the js-error-handler-auditor specializes in. Launch it to perform the audit.\n</commentary>\n</example>\n\n<example>\nContext: User is preparing for a code review and wants to ensure error handling best practices.\nuser: "I want to make sure our codebase follows error handling best practices before the review tomorrow."\nassistant: "I'll launch the js-error-handler-auditor agent to scan your JavaScript codebase for any unhandled errors in async operations and event handlers."\n<commentary>\nThis is a perfect use case for proactive error handling audit before code review.\n</commentary>\n</example>\n\n<example>\nContext: User just finished adding several new features with async code.\nuser: "I just added a bunch of new API calls and event listeners. Can you review them?"\nassistant: "I'll use the js-error-handler-auditor agent to analyze the error handling in your new async code and event listeners."\n<commentary>\nNew async code is prone to missing error handlers, making this an ideal time to run the audit agent.\n</commentary>\n</example>
+model: sonnet
+color: red
+---
+
+You are an elite JavaScript error handling auditor with deep expertise in async programming patterns, event-driven architectures, and production-grade error management. Your mission is to identify and report all instances of inadequate error handling in JavaScript codebases, focusing on top-level operations that commonly lack proper error boundaries.
+
+## Your Systematic Audit Process
+
+### Phase 1: Comprehensive Code Discovery
+
+First, you will search the entire codebase for common patterns that require error handling. Use the Bash tool to execute this search command:
+
+```bash
+rg -n --type js --type jsx --type ts --type tsx -e 'setTimeout|setInterval|addEventListener|Promise\.(?:all|race|allSettled|any)|fetch\(|async\s+function|await\s+|on\(|once\(|\.then\(|\.catch\(' . 2>/dev/null || echo "No matches found"
+```
+
+This command searches for:
+- `setTimeout` and `setInterval` - timer callbacks that may throw
+- `addEventListener`, `on()`, `once()` - event handlers
+- `Promise.all`, `Promise.race`, etc. - promise combinators
+- `fetch()` - network requests
+- `async function` and `await` - async/await patterns
+- `.then()` and `.catch()` - promise chains
+
+Capture all file paths and line numbers from this search.
+
+### Phase 2: Detailed Code Analysis
+
+For each location identified:
+
+1. **Read the full context** (at least 20 lines before and after) using the Read tool to understand:
+   - The complete function or block containing the pattern
+   - Existing error handling mechanisms (try/catch, .catch(), error callbacks)
+   - The broader context and error propagation paths
+
+2. **Evaluate error handling quality** by checking:
+   - **Async Functions**: Do they have try/catch blocks or are they wrapped in error boundaries?
+   - **Promises**: Do `.then()` chains have corresponding `.catch()` handlers?
+   - **Event Handlers**: Do `addEventListener` and event emitter callbacks have internal error handling?
+   - **Timers**: Do `setTimeout`/`setInterval` callbacks handle errors or are they wrapped?
+   - **Top-level awaits**: Are they in try/catch blocks or do they have .catch() handlers?
+   - **Promise combinators**: Are `Promise.all()`, etc. followed by `.catch()` or in try/catch?
+
+3. **Classify the severity**:
+   - **Critical**: Unhandled async operations at module/component initialization level
+   - **High**: User-triggered actions (click handlers, form submissions) without error handling
+   - **Medium**: Background operations (timers, polling) without error handling
+   - **Low**: Promise chains with partial error handling
+
+### Phase 3: Comprehensive Reporting
+
+After analyzing all locations, provide a structured report with:
+
+1. **Executive Summary**
+   - Total locations scanned
+   - Number of issues found by severity
+   - Overall error handling score (percentage of properly handled cases)
+
+2. **Detailed Issue List**
+
+For each unhandled or improperly handled error location, provide:
+
+```
+## Issue #N: [Severity] [Pattern Type]
+
+**Location**: `filepath:line_number`
+
+**Current Code**:
+```javascript
+[Exact code snippet showing the problematic pattern]
+```
+
+**Problem**: [Clear explanation of why this is problematic and what could go wrong]
+
+**Recommended Fix**:
+```javascript
+[Complete, production-ready code showing proper error handling]
+```
+
+**Explanation**: [Why this fix is appropriate and what it accomplishes]
+```
+
+### Your Error Handling Expertise
+
+When evaluating code, apply these expert principles:
+
+**Valid Error Handling Patterns**:
+- Async functions wrapped in try/catch
+- Promise chains with `.catch()` handlers
+- Event handlers with internal try/catch
+- Error boundaries in appropriate contexts
+- Explicit error callbacks in Node.js patterns
+- Global error handlers (window.onerror, unhandledrejection) as last resort
+
+**Invalid or Insufficient Patterns**:
+- "Fire and forget" async calls without error handling
+- Promise chains ending in `.then()` without `.catch()`
+- Event handlers that can throw but lack error handling
+- Async functions called synchronously without await or .catch()
+- Timer callbacks that perform async operations without error handling
+
+**Context Matters**:
+- Consider if errors are intentionally propagating to a higher-level handler
+- Recognize when logging is insufficient (errors should be handled, not just logged)
+- Identify when errors need user feedback vs. silent handling
+- Understand framework-specific error handling (React error boundaries, Vue error handlers, etc.)
+
+### Output Format
+
+Always structure your final report as:
+
+1. Executive summary with statistics
+2. Numbered list of all issues, ordered by severity (Critical → High → Medium → Low)
+3. Each issue must include: location, current code, problem description, recommended fix, and explanation
+4. Conclude with general recommendations for improving error handling practices in the codebase
+
+### Quality Assurance
+
+Before presenting your report:
+- Verify you've analyzed every location from the initial search
+- Ensure all code snippets are accurate and complete
+- Confirm all recommended fixes are syntactically correct and idiomatic
+- Double-check that you haven't flagged code that already has adequate error handling
+- Validate that your severity classifications are appropriate
+
+You are thorough, precise, and your recommendations must be immediately actionable by developers. Every issue you report must be a genuine error handling gap, and every fix you propose must be production-ready.

biome.json

@@ -1,5 +1,5 @@
 {
-	"$schema": "https://biomejs.dev/schemas/2.1.1/schema.json",
+	"$schema": "https://biomejs.dev/schemas/2.2.3/schema.json",
 	"files": {
 		"includes": [
 			"**/*.js",
@@ -41,6 +41,10 @@
 			},
 			"suspicious": {
 				"noExplicitAny": "off"
+			},
+			"nursery": {
+				"noFloatingPromises": "error",
+				"noMisusedPromises": "error"
 			}
 		}
 	},

engine/sdks/typescript/runner/src/mod.ts

@@ -8,6 +8,7 @@ import { type HibernatingWebSocketMetadata, Tunnel } from "./tunnel";
 import {
 	calculateBackoff,
 	parseWebSocketCloseReason,
+	stringifyError,
 	unreachable,
 } from "./utils";
 import { importWebSocket } from "./websocket.js";
@@ -23,6 +24,12 @@ const PROTOCOL_VERSION: number = 4;
 const EVENT_BACKLOG_WARN_THRESHOLD = 10_000;
 const SIGNAL_HANDLERS: (() => void | Promise<void>)[] = [];
 
+export class RunnerShutdownError extends Error {
+	constructor() {
+		super("Runner shut down");
+	}
+}
+
 export interface RunnerConfig {
 	logger?: Logger;
 	version: number;
@@ -250,7 +257,14 @@ export class Runner {
 
 		// Start cleaning up old unsent KV requests every 15 seconds
 		this.#kvCleanupInterval = setInterval(() => {
-			this.#cleanupOldKvRequests();
+			try {
+				this.#cleanupOldKvRequests();
+			} catch (err) {
+				this.log?.error({
+					msg: "error cleaning up kv requests",
+					error: stringifyError(err),
+				});
+			}
 		}, 15000); // Run every 15 seconds
 	}
 
@@ -309,11 +323,7 @@ export class Runner {
 
 		// Remove all remaining kv requests
 		for (const [_, request] of this.#kvRequests.entries()) {
-			request.reject(
-				new Error(
-					"KV request timed out waiting for WebSocket connection",
-				),
-			);
+			request.reject(new RunnerShutdownError());
 		}
 
 		this.#kvRequests.clear();
@@ -324,7 +334,13 @@ export class Runner {
 	#stopAllActors() {
 		const actorIds = Array.from(this.#actors.keys());
 		for (const actorId of actorIds) {
-			this.forceStopActor(actorId);
+			this.forceStopActor(actorId).catch((err) => {
+				this.log?.error({
+					msg: "error stopping actor",
+					actorId,
+					error: stringifyError(err),
+				});
+			});
 		}
 	}
 
@@ -761,12 +777,19 @@ export class Runner {
 			// Start command acknowledgment interval (5 minutes)
 			const ackInterval = 5 * 60 * 1000; // 5 minutes in milliseconds
 			const ackLoop = setInterval(() => {
-				if (ws.readyState === 1) {
-					this.#sendCommandAcknowledgment();
-				} else {
-					clearInterval(ackLoop);
-					this.log?.info({
-						msg: "WebSocket not open, stopping ack loop",
+				try {
+					if (ws.readyState === 1) {
+						this.#sendCommandAcknowledgment();
+					} else {
+						clearInterval(ackLoop);
+						this.log?.info({
+							msg: "WebSocket not open, stopping ack loop",
+						});
+					}
+				} catch (err) {
+					this.log?.error({
+						msg: "error in command acknowledgment loop",
+						error: stringifyError(err),
 					});
 				}
 			}, ackInterval);
@@ -826,12 +849,17 @@ export class Runner {
 				const kvResponse = message.val;
 				this.#handleKvResponse(kvResponse);
 			} else if (message.tag === "ToClientTunnelMessage") {
-				this.#tunnel?.handleTunnelMessage(message.val);
+				this.#tunnel?.handleTunnelMessage(message.val).catch((err) => {
+					this.log?.error({
+						msg: "error handling tunnel message",
+						error: stringifyError(err),
+					});
+				});
 			} else if (message.tag === "ToClientPing") {
 				this.__sendToServer({
 					tag: "ToServerPong",
 					val: {
-						ts: message.val.ts
+						ts: message.val.ts,
 					},
 				});
 			} else {
@@ -856,7 +884,14 @@ export class Runner {
 						seconds: this.#runnerLostThreshold / 1000,
 					});
 					this.#runnerLostTimeout = setTimeout(() => {
-						this.#handleLost();
+						try {
+							this.#handleLost();
+						} catch (err) {
+							this.log?.error({
+								msg: "error handling runner lost",
+								error: stringifyError(err),
+							});
+						}
 					}, this.#runnerLostThreshold);
 				}
 
@@ -912,7 +947,14 @@ export class Runner {
 						seconds: this.#runnerLostThreshold / 1000,
 					});
 					this.#runnerLostTimeout = setTimeout(() => {
-						this.#handleLost();
+						try {
+							this.#handleLost();
+						} catch (err) {
+							this.log?.error({
+								msg: "error handling runner lost",
+								error: stringifyError(err),
+							});
+						}
 					}, this.#runnerLostThreshold);
 				}
 
@@ -931,29 +973,52 @@ export class Runner {
 		for (const commandWrapper of commands) {
 			if (commandWrapper.inner.tag === "CommandStartActor") {
 				// Spawn background promise
-				this.#handleCommandStartActor(commandWrapper);
+				this.#handleCommandStartActor(commandWrapper).catch((err) => {
+					this.log?.error({
+						msg: "error handling start actor command",
+						actorId: commandWrapper.checkpoint.actorId,
+						error: stringifyError(err),
+					});
+				});
 			} else if (commandWrapper.inner.tag === "CommandStopActor") {
 				// Spawn background promise
-				this.#handleCommandStopActor(commandWrapper);
+				this.#handleCommandStopActor(commandWrapper).catch((err) => {
+					this.log?.error({
+						msg: "error handling stop actor command",
+						actorId: commandWrapper.checkpoint.actorId,
+						error: stringifyError(err),
+					});
+				});
 			} else {
 				unreachable(commandWrapper.inner);
 			}
 
-			const actor = this.getActor(commandWrapper.checkpoint.actorId, commandWrapper.inner.val.generation);
+			const actor = this.getActor(
+				commandWrapper.checkpoint.actorId,
+				commandWrapper.inner.val.generation,
+			);
 			if (actor) actor.lastCommandIdx = commandWrapper.checkpoint.index;
 		}
 	}
 
 	#handleAckEvents(ack: protocol.ToClientAckEvents) {
-		let originalTotalEvents = Array.from(this.#actors).reduce((s, [_, actor]) => s + actor.eventHistory.length, 0);
+		let originalTotalEvents = Array.from(this.#actors).reduce(
+			(s, [_, actor]) => s + actor.eventHistory.length,
+			0,
+		);
 
 		for (const [_, actor] of this.#actors) {
-			let checkpoint = ack.lastEventCheckpoints.find(x => x.actorId == actor.actorId);
+			let checkpoint = ack.lastEventCheckpoints.find(
+				(x) => x.actorId == actor.actorId,
+			);
 
 			if (checkpoint) actor.handleAckEvents(checkpoint.index);
 		}
 
-		const totalEvents = Array.from(this.#actors).reduce((s, [_, actor]) => s + actor.eventHistory.length, 0);
+		const totalEvents = Array.from(this.#actors).reduce(
+			(s, [_, actor]) => s + actor.eventHistory.length,
+			0,
+		);
 		const prunedCount = originalTotalEvents - totalEvents;
 
 		if (prunedCount > 0) {
@@ -975,7 +1040,10 @@ export class Runner {
 
 		actor.recordEvent(eventWrapper);
 
-		let totalEvents = Array.from(this.#actors).reduce((s, [_, actor]) => s + actor.eventHistory.length, 0);
+		let totalEvents = Array.from(this.#actors).reduce(
+			(s, [_, actor]) => s + actor.eventHistory.length,
+			0,
+		);
 
 		if (
 			totalEvents > EVENT_BACKLOG_WARN_THRESHOLD &&
@@ -1708,13 +1776,18 @@ export class Runner {
 			msg: `Scheduling reconnect attempt ${this.#reconnectAttempt + 1} in ${delay}ms`,
 		});
 
-		this.#reconnectTimeout = setTimeout(async () => {
+		this.#reconnectTimeout = setTimeout(() => {
 			if (!this.#shutdown) {
 				this.#reconnectAttempt++;
 				this.log?.debug({
 					msg: `Attempting to reconnect (attempt ${this.#reconnectAttempt})...`,
 				});
-				await this.#openPegboardWebSocket();
+				this.#openPegboardWebSocket().catch((err) => {
+					this.log?.error({
+						msg: "error during websocket reconnection",
+						error: stringifyError(err),
+					});
+				});
 			}
 		}, delay);
 	}