LLMs.txt is also nonsense since it isn't adopted by any of the major AI players.
The same could be said of robots.txt
And anything else that might tell them not to access something.
To be fair, "not adopted by any major AI player" is probably the most web-standard-compliant phase of a new web standard.
Google has recently added `llms.txt` to Chrome's Lighthouse check for agentic browsing (https://searchengineland.com/google-llms-txt-chrome-lighthou...), so adoption may be coming. Admittedly, I put more faith in
that I copied from Gwern.net. This convention is discoverable (just read the HTML) and naturally adapts to any website size and structure.I have created an `llms.txt` for my website anyhow. I use a fixed LLM prompt to generate it from the internal links in `index.md`.