Can llms.txt Control AI Model Access to My Site?

The short answer: no

llms.txt cannot control which AI models or crawlers access your site. It is a proposed convention (llmstxt.org, September 2024) for a Markdown file at /llms.txt that describes your site and links its most useful pages. It is a reading guide, not a gate. Nothing in the format allows or denies a crawler, and no major AI engine has said it treats the file as a permission signal.

If you want to decide who may fetch your content, and for what purpose, you need the tools built for that job. The rest of this page covers what they are and how to use them without accidentally disappearing from AI answers.

The confusion is common because the names look alike. robots.txt has governed crawling for decades, so a file called llms.txt sounds like its AI equivalent. It is not, and treating it as one leaves sites either unprotected or, more often, invisible without anyone knowing why.

What actually controls AI access

• robots.txt: per-user-agent rules that well-behaved crawlers, including the major AI ones, follow. Advisory, but widely respected.
• CDN and firewall bot rules: enforced at the network level, able to allow, challenge or block requests regardless of robots.txt.
• Logins and paywalls: content behind authentication is not fetched by public crawlers.
• Page-level controls such as noindex, nosnippet, data-nosnippet and max-snippet, which govern how search engines, including Google's AI features, may use a page.

The gap between the first two is where most problems hide. A robots.txt that allows OAI-SearchBot means nothing if the CDN's bot protection challenges every unfamiliar user agent. Many sites are invisible to AI search for this reason and do not know it, because no standard report shows it.

Citation crawlers versus training crawlers

AI companies run different crawlers for different purposes, and they deserve different decisions. Citation crawlers such as OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot fetch pages to answer live questions and link to sources. Blocking them generally means those engines cannot cite you. Training crawlers such as GPTBot and ClaudeBot gather content for model training, which is a separate policy and commercial choice.

Google is a special case. Google-Extended is a robots.txt token that controls whether your content is used for Gemini training and grounding. It does not affect inclusion in Google Search or AI Overviews, which rely on Googlebot and follow normal Search rules.

Write the decision down. A short internal policy listing each AI user agent, its purpose and your choice makes later reviews easy, and stops a well-meant security change from blocking the crawlers that bring you citations.

How to check your access setup properly

Start with robots.txt and write down the rule that applies to each AI user agent. Then test at the network level: request key pages using each AI crawler's user agent, look for challenges, blocks or empty responses, and review the bot management settings at your CDN or firewall. Server or CDN logs show which AI crawlers actually reach you and which receive errors.

Finally, check that important facts are in the HTML itself, not only rendered by JavaScript, since some crawlers do not execute scripts. An open door is no use if the room looks empty to the visitor.

Worked example: a broker that blocked itself

Picture a forex broker that wants to appear when traders ask ChatGPT which brokers are licensed in their country. Its robots.txt allows all AI citation crawlers, and its team has published an llms.txt. Yet ChatGPT search never cites its pages.

An audit of a case like this typically traces the cause to the CDN: a bot-protection mode switched on after a scraping incident challenges any non-browser user agent, including OAI-SearchBot and PerplexityBot. The fix is an allowlist for verified citation crawlers that keeps protection against scrapers in place. The llms.txt was never the problem and could never have been the fix. This is illustrative, not a client result.

The same pattern turns up on software, legal and industrial sites, wherever a security team has tightened bot rules without a list of the crawlers the business actually wants. Access fixes are usually among the cheapest wins in a whole programme.

What Suggesting.ai checks and fixes

Suggesting.ai's free audit, delivered in 48 hours, starts with access: robots.txt rules for each AI user agent, network-level tests that reveal CDN and firewall blocks robots.txt does not show, and rendering checks. It then records live answers on ChatGPT, Perplexity, Google AI features and Copilot, competitors named, sources cited, content readiness and entity consistency.

From there we run organic GEO and, where your category and market are eligible, ChatGPT Ads. For forex brokers and other financial services outside the US, OpenAI's policy generally does not permit ads, so open access and earned citations carry the load. Before AI can start suggesting you, it has to be able to read you.

Monitoring access over time

Access breaks quietly, often after a CDN change or a security incident. Review AI crawler hits and error rates in your logs weekly, re-run user-agent tests after any infrastructure change, and track citation share on your buyer questions monthly. A sudden drop in citations on one engine is often an access problem before it is a content problem.

Assign an owner. Access settings usually sit with engineering or security, while the impact shows up in marketing, so the person watching citations needs a direct line to whoever manages the CDN.