> Why isn't it possible to limit it to a specific whitelisted set?

It is but you need to use an MCP client and servers that implement Roots.