mcp: Add OAuth 2.1 support in MCPTool/MCPToolset

[vblagoje]

Objective

Implement complete OAuth 2.1 support in the Haystack MCP Tool integration per MCP Authorization Specification, covering both interactive (Authorization Code + PKCE) flow and automated (Client Credentials) flow. This includes automatic discovery via WWW-Authenticate, optional dynamic client registration, local HTTP callback handling, and restricting OAuth to HTTP transports.

Scope clarification

This work aims only to implement OAuth client behavior in Haystack MCPTool.

We are not:

• Implementing an OAuth 2.1 authorization server
• Providing an SDK for adding OAuth to MCP servers themselves
• Remote MCP servers are assumed to already expose standards-compliant OAuth 2.1 / MCP auth endpoints.

Current State

• OAuth is not currently implemented in the Haystack MCP integration
• The integration currently supports only basic token-based authentication via headers (token parameter in StreamableHttpServerInfo and SSEServerInfo)

Architecture Context

The Haystack MCP integration uses the following key abstractions:

• MCPServerInfo (abstract base): StreamableHttpServerInfo, SSEServerInfo, StdioServerInfo
• MCPClient (abstract base): StreamableHttpClient, SSEClient, StdioClient
• MCPTool: Single tool from an MCP server, extends Haystack's Tool class
• MCPToolset: Collection of tools from an MCP server, extends Haystack's Toolset class

Integration Points

OAuth should integrate at these key points:

1. MCPTool.__init__() / MCPToolset.__init__(): If eager_connect=True, OAuth should happen during initialization
2. MCPTool.warm_up() / MCPToolset.warm_up(): If eager_connect=False, OAuth should happen during warm-up (before first connection)
3. StreamableHttpClient.connect(): Catch 401 responses and trigger OAuth discovery
4. StreamableHttpClient.call_tool(): Catch 401 responses during tool invocations and retry with token
5. MCPTool._connect_and_initialize() / MCPToolset._connect_and_load_tools(): OAuth should be checked before connection

Deliverables

• ✅ Dual support for Authorization Code + PKCE and Client Credentials grant types
• ✅ Automatic detection of 401 with WWW-Authenticate, discovery of auth endpoints via metadata
• ✅ Optional automatic localhost HTTP callback with fallback to manual for Authorization Code flow
• ✅ Enforcement of HTTP-only OAuth support (no OAuth for STDIO transport)
• ✅ Config API updates to support flexible flow selection (auto + manual)
• ✅ Full serialization support for OAuthConfig in MCPServerInfo.to_dict()/from_dict()
• ✅ Token storage with persistence across tool instances (file-based and in-memory)
• ✅ Non-invasive integration - existing code works without OAuth

📌 This issue will track the overall effort. See linked subtasks for implementation milestones.

Sub-issues

• mcp: Implement OAuth 2.1 Grant Flows (Authorization Code + PKCE and Client Credentials) #2546 - Implement OAuth 2.1 Grant Flows (Authorization Code + PKCE and Client Credentials)
• mcp: Implement Automatic OAuth Callback Handling with Local Server #2547 - Implement Automatic OAuth Callback Handling with Local Server
• mcp: Implement Automatic OAuth Discovery via 401 Detection #2548 - Implement Automatic OAuth Discovery via 401 Detection