mcp: Implement Automatic OAuth Discovery via 401 Detection

[vblagoje]

Scope

Implement automatic OAuth discovery when MCP servers return 401 Unauthorized responses, following the MCP spec's discovery mechanisms. This enables zero-configuration OAuth for compliant servers.

Haystack MCP Integration Details

401 Detection Points:

• StreamableHttpClient.connect(): Catch 401 during initial connection/session establishment
• StreamableHttpClient.call_tool(): Catch 401 during tool invocations
• Both should trigger the same discovery and OAuth flow

Discovery Flow:

1. Parse WWW-Authenticate: Bearer ... header to extract metadata hints
2. Try WWW-Authenticate header resource_metadata parameter first
3. Fallback to /.well-known/oauth-protected-resource{path} (with path insertion per RFC 9728)
4. Fallback to /.well-known/oauth-protected-resource (root)
5. Extract authorization_servers list from resource metadata
6. Fetch authorization server metadata from /.well-known/oauth-authorization-server
7. Extract authorization_endpoint, token_endpoint, registration_endpoint, etc.
8. Validate S256 PKCE support (required by MCP spec)

Dynamic Client Registration (DCR):

• Check for registration_endpoint in AS metadata
• POST to register client and obtain client_id (and client_secret if confidential)
• Fallback to requiring user-provided client_id if DCR not supported
• This enables truly zero-config OAuth for servers that support it

Auto-Population of OAuthConfig:

• Create OAuthConfig instance from discovered metadata
• Fill in authorization_endpoint, token_endpoint from metadata
• If DCR succeeds, populate client_id and client_secret
• Store discovered config for reuse (avoid repeated discovery)

Retry Logic:

• After obtaining token, retry the original request that triggered 401
• Cache token in server_info.headers["Authorization"] for subsequent requests
• Handle token expiration and refresh (if refresh token available)

Integration with Existing Code:

• Discovery should be non-invasive - existing code should work without OAuth
• If OAuthConfig is provided, use it (still perform discovery for endpoints if not specified)
• Discovery should happen lazily (only on 401) to avoid unnecessary requests
• Consider caching discovered metadata to avoid repeated discovery

Error Handling:

• If discovery fails, provide clear error messages
• If DCR fails, fallback to requiring user-provided credentials
• If OAuth flow fails, propagate meaningful errors

Requirements

• Catch 401 Unauthorized responses in StreamableHttpClient.connect() and call_tool()
• Parse WWW-Authenticate: Bearer ... header to extract metadata hints
• Implement automatic resource metadata discovery:
  • Try WWW-Authenticate header resource_metadata parameter first
  • Fallback to /.well-known/oauth-protected-resource{path} (with path insertion)
  • Fallback to /.well-known/oauth-protected-resource (root)
• Implement automatic authorization server metadata discovery:
  • Fetch /.well-known/oauth-authorization-server from discovered issuer
  • Extract authorization_endpoint, token_endpoint, code_challenge_methods_supported
  • Validate S256 PKCE support (required by MCP spec)
• (Optional) Implement Dynamic Client Registration (RFC 7591):
  • Check for registration_endpoint in AS metadata
  • POST to register client and obtain client_id (and client_secret if confidential)
  • Fallback to requiring user-provided client_id if DCR not supported
• Auto-populate OAuthConfig from discovered metadata
• Trigger appropriate OAuth flow (Authorization Code or Client Credentials) based on discovery
• Retry original request after obtaining token
• Cache discovered metadata and tokens to avoid repeated discovery
• Handle errors gracefully with clear messages
• Add comprehensive tests for discovery flow
• Add tests for 401 handling and retry logic
• Add tests for DCR (if implemented)

References

• MCP Authorization Spec - Discovery
• RFC 9728 - OAuth 2.0 Protected Resource Metadata
• RFC 8414 - OAuth 2.0 Authorization Server Metadata
• RFC 7591 - OAuth 2.0 Dynamic Client Registration