llms.txt Guidelines for Publishers and Content-Heavy Websites
Publishers should keep llms.txt short and curated: a clear description of the publication and links to section hubs, evergreen explainers, data pages, editorial standards and a licensing contact, not the whole archive. It cannot decide which AI companies may crawl or train on your content; robots.txt, CDN or firewall rules and contracts do. No major AI engine has confirmed using llms.txt for citations, so treat it as hygiene and put the effort into being citable.
Why publishers face a different llms.txt question
For a publisher, llms.txt raises a sharper question than for most businesses. Your content is the product, and AI answers can satisfy a reader without a visit. So the decision is not only how to summarise the site, but how to be cited in AI answers while keeping control over how your work is used.
llms.txt, a proposed convention published at llmstxt.org in September 2024, only addresses the first part. It is a Markdown file at /llms.txt that describes your site and links its key pages. It has no say over access or use, and no major AI engine has confirmed it uses the file for ranking or citations. Google has said Search does not use it.
That leaves llms.txt in a modest role: a clear, curated pointer to your best work for the tools that read it. Its real value for publishers is the editorial discipline of deciding which pages represent the publication, and making sure those pages are the ones AI engines can reach and quote.
Curating an archive into a short file
A site with years of articles cannot list them all, and should not try. Curate in layers:
- Section hubs for each beat or topic, with one sentence on what each covers.
- Evergreen explainers and glossaries that answer recurring reader questions.
- Data pages, tools and calculators that are frequently referenced.
- Editorial standards, corrections policy, ownership and author pages.
- A licensing or syndication contact for AI companies and partners.
Leave out daily news URLs, tag pages, paginated archives and anything behind a hard paywall. If you publish a longer companion file with full text, keep it to evergreen material and weigh its size and update cost before committing to it.
For very large sites, keep the root file short and link to separate summary pages maintained by each desk. Ownership stays with the editors who know the material, and the root file stays readable.
| Decision | Where it is controlled | Can llms.txt do it? |
|---|---|---|
| Allow AI search engines to cite you | robots.txt and CDN or firewall rules | No |
| Block or allow model training | robots.txt for training crawlers, plus contracts | No |
| Limit what Google shows from a page | nosnippet, data-nosnippet, max-snippet | No |
| Point AI tools to your best content | llms.txt, sitemaps, internal links | Yes, for tools that read it |
| License content to AI companies | Contracts and licensing terms | It can only list a contact |
Decisions llms.txt cannot make for you
Publishers need a clear crawler policy, and it lives elsewhere. Citation crawlers such as OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot fetch pages to answer live questions and cite sources; blocking them usually removes you from those answers. Training crawlers such as GPTBot and ClaudeBot collect content for model training, which is a separate commercial decision. Google-Extended controls use of your content for Gemini training and grounding, not inclusion in Search or AI Overviews.
These choices are expressed in robots.txt and enforced, where needed, at the CDN or firewall. Licensing terms belong in contracts. llms.txt can name your licensing contact, but it cannot grant or deny anything.
Making articles citable once crawlers can reach them
Being cited depends on whether an engine can lift a clear answer from your page. Put the direct answer near the top of explainers, use descriptive headings, date updates visibly and name authors with real credentials. Keep data in HTML tables rather than images.
For Google's AI features, the usual fundamentals apply: indexable pages, helpful content and structured data that matches the visible text. Preview controls such as nosnippet, data-nosnippet and max-snippet let you limit what Google shows from a page, at the cost of some visibility. Use them deliberately on premium content, not as a blanket setting across the site.
| Section | What to link | Update cadence |
|---|---|---|
| Summary | Who you are, beats covered, languages | When scope changes |
| Section hubs | One hub per topic with a one-line description | Quarterly |
| Evergreen | Explainers, glossaries, data pages | Monthly |
| Trust | Editorial standards, corrections, authors, ownership | When policies change |
| Licensing | Contact for syndication and AI use | As needed |
Worked example: a financial news publisher
Consider a finance publisher covering markets, brokers and trading education in several languages. Its llms.txt would link hubs for forex, commodities and stocks, a broker reviews hub, a glossary, its methodology for broker ratings, editorial standards and a licensing contact, with a separate file for each language edition.
The higher-value work sits around the file: allowing citation crawlers at the CDN while deciding separately on training crawlers, making broker review pages state licence and fee facts in plain text, and keeping author and ownership information consistent across editions. Because broker content touches regulated financial promotion, accuracy and clear risk wording matter to readers and engines alike.
Multilingual publishers face one extra trap: translated pages that are not clearly connected. Correct hreflang annotations and consistent author and ownership pages help engines understand that every edition belongs to the same trusted publisher.
What Suggesting.ai does for publishers
Suggesting.ai works with finance and trading media brands, so publisher questions are familiar ground. The free audit, delivered in 48 hours, tests the questions your readers ask AI, records whether your articles are cited and which rivals are, checks crawler access including CDN and firewall blocks that robots.txt does not reveal, and reviews entity consistency and your llms.txt.
We then run organic GEO on citability and coverage, and can manage ChatGPT Ads for subscriptions or products where your category and market are eligible. For publishers promoting financial products outside the US, ad eligibility is generally restricted and is confirmed in the audit. When an assistant is suggesting sources on your beat, you want to be the one it quotes.
Measuring citations, not just visits
Track how often your pages are cited for a fixed set of reader questions, by engine and by section. In analytics, segment AI referrals: ChatGPT links carry utm_source=chatgpt.com, and Perplexity, Gemini and Copilot appear partly as referrers. Check server logs for /llms.txt requests by user agent. Because many AI answers are read without a click, citation share is the leading indicator; sessions follow.
Review the crawler policy at least twice a year, since new AI user agents appear and business terms change.
Frequently asked questions
Should publishers list every article in llms.txt?
No. Link section hubs, evergreen explainers, data pages and trust pages. Daily news URLs lose relevance too quickly, and archives are better handled by sitemaps. A short, curated file is easier to keep accurate and more useful to the tools that read it than an exhaustive list.
Can llms.txt stop AI companies training on my articles?
No. llms.txt is informational. Training crawlers such as GPTBot and ClaudeBot are addressed in robots.txt, with enforcement possible at the CDN or firewall, and commercial terms belong in licensing agreements. Google-Extended covers Gemini training and grounding, separately from Google Search.
Will blocking AI crawlers protect my traffic?
Blocking citation crawlers usually removes your articles from AI answers, which can cost visibility and referrals rather than protect them. Many publishers allow citation crawlers and make a separate decision on training crawlers. The right balance depends on your business model and any licensing plans.
Do publishers need a separate llms.txt per language?
If each edition lives on its own subdomain or path with distinct content, a file per edition is clearer, written in that language and linking that edition's hubs. Keep ownership, author and editorial information consistent across editions, so engines see one coherent publisher rather than several unrelated sites.
Does Google use llms.txt for AI Overviews?
Google has said Search does not use llms.txt, and its guidance for AI Overviews and AI Mode says no special file or markup is needed. The same Search fundamentals apply: crawlable, indexable pages, helpful content and structured data that matches the visible text on the page.
Want AI to suggest your brand instead of a competitor?
See which publishers AI engines cite on your beats, and why: request Suggesting.ai's free 48-hour AI visibility audit for your site.