Auditing Your llms.txt, and What Else Decides AI Discoverability

What auditing llms.txt can and cannot achieve

Auditing your llms.txt is worth an hour. The file, a proposed convention published at llmstxt.org in September 2024, is a Markdown summary at /llms.txt that links your key pages for language models. A clean one helps the AI tools and agents that read it, and the audit often exposes wider problems: broken links, outdated facts or important pages nobody can find.

Set expectations first. llms.txt does not control which crawlers reach your site, and no major AI engine has confirmed using it for ranking or citations; Google has said Search does not use it. A perfect file will not, on its own, improve AI discoverability. That is why a useful llms.txt audit always widens into an access and content audit.

The steps below take an hour or two for most sites and need nothing more than a browser, a command line and access to your CDN settings.

Step one: format and location

• The file is served at the root, /llms.txt, with a 200 status and a plain-text content type.
• It starts with an H1 title, followed by a short summary in a blockquote, as the llmstxt.org format describes.
• Links sit under H2 headings, each as a Markdown link followed by a short description.
• Secondary material is grouped under a clearly labelled optional section.
• The file is not blocked by robots.txt or by a CDN rule for non-browser user agents.

That last point matters more than it looks. If your firewall challenges unfamiliar user agents, the tools that would read llms.txt cannot fetch it, and neither can the AI crawlers that matter far more.

Also confirm the file behaves the same on every host variant you use, including www and non-www and any country subdomains. Redirects are fine if they resolve in one hop to the canonical version.

Step two: content and accuracy

Read the file as a buyer would. Does the summary say plainly what you sell, to whom and where? Are the linked pages the ones that answer buying questions: products, pricing, comparisons, case studies, documentation and trust pages such as licences or certifications? Is each description factual rather than promotional?

Then check every link. Each should return 200, be indexable, avoid redirects and not be blocked anywhere. Compare the facts in the file with the linked pages, your sitemap and your public profiles. A different company name, an old product name or a retired pricing tier points to an entity consistency problem that probably exists across the web, not just in this file.

Check coverage too. If your most-asked buyer questions are answered on pages the file does not link, either add them or ask why those pages are not prominent on the site itself. Gaps in llms.txt often mirror gaps in your navigation.

Step three: widen the audit to what drives discoverability

Once the file is clean, test what decides whether AI engines find and cite you. Check access for each AI citation crawler, namely OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot, at the network level, because CDN and firewall rules often block what robots.txt allows. Confirm that Googlebot can crawl and index your key pages, since Google's AI features rely on normal Search.

Then run your buyer questions on ChatGPT, Perplexity, Google AI features and Copilot. Record whether you appear, which competitors are named, which sources are cited and whether your facts are correct. This is where discoverability is actually decided.

Finally, check entity consistency outside your own site: business listings, review platforms, partner directories and, for regulated firms, the regulator's public register. Engines combine these sources, and a mismatch anywhere can surface in an answer.

Worked example: auditing a broker's llms.txt

Imagine a forex broker's llms.txt that links an expired promotions page, a platform page that now redirects, and a licensing page describing an entity the broker restructured last year. The summary calls the broker the best in its region, a claim no engine will repeat and the local regulator may object to.

The fixes in the file are quick: replace the promotion with account-type and fees pages, update the licensing description to match the regulator's register and drop the superlative. The wider audit matters more. It might find that the broker's CDN challenges PerplexityBot and that comparison sites cited in answers still list the old entity name. This illustrates the method; it is not a client result.

What Suggesting.ai includes

Suggesting.ai's free audit, delivered in 48 hours, reviews your llms.txt against the checks above and then covers what moves AI answers: crawler access including CDN and firewall blocks robots.txt does not show, live answers on ChatGPT, Perplexity, Google AI and Copilot, competitors named, sources cited, content readiness and entity consistency.

Follow-on work combines organic GEO with ChatGPT Ads where your category and market qualify. For financial services outside the US, OpenAI's policy generally rules ads out, and the audit says so up front. A tidy file is a start; getting AI to suggest you is the job.

Tracking whether discoverability improves

Log requests to /llms.txt by user agent to see whether anything reads it. For discoverability itself, re-run the same buyer questions monthly and track mention rate, citation share and accuracy, then connect them to AI referral sessions, where ChatGPT links carry utm_source=chatgpt.com, and to the leads those sessions produce.

Re-audit the file itself quarterly, or whenever you launch, rename or retire a product, so it never drifts out of step with the live site.