What the generator does
It reads your robots.txt, finds your sitemap if you publish one, and fetches up to ten pages. For each page it keeps the URL, the title and the meta description, then sorts them into product, pricing, documentation, writing, company and legal by their path.
The output is one markdown file. Pages that returned anything other than a 200 are excluded and listed underneath with the reason, so a gap in the index is a decision you can see rather than a silent omission.
Where to put the file
Save it as llms.txt at the root of your domain, next to robots.txt, so it resolves at yoursite.com/llms.txt. Most frameworks serve anything in the public or static directory from the root; on a CMS it is usually a redirect or a file upload rather than a page.
It is worth publishing only if an assistant can reach your pages in the first place, so run the AI crawler access check as well. A perfect index behind a blanket robots.txt block helps nobody.
How often to regenerate it
Whenever the shape of the site changes: a new section, a renamed pricing page, a documentation move. The file is an index, so the failure mode is staleness rather than wrongness, and a stale index is worse than none once it starts pointing at pages that have moved.
Keep reading
Last updated August 31, 2026. All 17 calculators and AI helpers are listed on the free tools hub.