Revisit tools approach

[swissspidy]

In the hackathon, we built a solution to automatically expose any REST API route as an MCP tool for function calling. This enabled us to quickly unlock a lot of use cases without any repetition. However, this also has some drawbacks:

1. We add a lot of tools out of the box, polluting the LLM's context
2. The input schemas for the tools are way too complex. Is the AI agent really ever going to construct a complex meta query? Probably not.
3. We send tools for every route, even if the user doesn't have the capability to access the data
4. The auto-generated descriptions are very generic (and not translatable), making it difficult for the LLM to understand the tool
   • At the hackathon we discussed that theoretically every route should have its proper description in core, but this ship has sailed

We should revisit this. An approach more similar to https://github.com/Automattic/wp-feature-api might make sense, where we manually define the tools for specific use cases. Still covering most CRUD operations, but with better quality.

See https://github.com/Automattic/wp-feature-api/blob/7d837ca7bbe40abbe7d7e4eed446e840fa90003c/includes/default-wp-features.php#L42-L79 and https://github.com/Automattic/wp-feature-api/blob/7d837ca7bbe40abbe7d7e4eed446e840fa90003c/includes/class-wp-feature.php#L730-L766 for inspiration.

What's still needed there is of course permissions checks for exposing the tool in the first place.

[mateuswetah]

The auto-generated descriptions are very generic (and not translatable), making it difficult for the LLM to understand the tool - At the hackathon we discussed that theoretically every route should have its proper description in core, but this ship has sailed

Might be off topic and a total trip from my head but Just to see if I'm getting this right... If in my own plugin, when registering the REST API routes to WordPress, I pass a schema (such as the json-schema that tools like ReDoc use) with proper descriptions... that same description may one day be usable by the LLM context to understand how to use it? 🤔

[swissspidy]

Correct, we already use that schema and the parameter descriptions. What WP lacks is a general description for the route itself.

[mateuswetah]

Hey @swissspidy just watched this and I believe you might be interested on the points made: https://youtu.be/eeOANluSqAE?si=_Z0Cy61BTrwLXTXw

Considering my plugin it is true that the OpenAPI schema does a nice first step on describing the usage but we should definitely be able to filter which endpoints to expose as tools and even modify or extend their meta