GEO Robots.txt Generator
Configure AI crawler access for optimal GEO visibility
📚 Educational Tool
This generates the AI crawler section of your robots.txt. Append to your existing file. Always test in staging first.
📖 Key Concepts Explained
Allow — Full access. The crawler can visit all pages (except blocked paths) at any speed. Best for maximum visibility.
Rate-Limit — Restricted speed. The crawler can visit pages but must wait X seconds between requests (Crawl-delay). Use for training crawlers that consume server resources.
Block — No access. The crawler cannot visit any pages. You become invisible to that AI system. Use for aggressive or unwanted crawlers.
Crawl-delay — Seconds the crawler must wait between page requests. Higher values = less server load but slower indexing. Typical: 2-10s for search crawlers, 10-30s for training crawlers.
User-agent: * — A wildcard rule that applies to ALL crawlers not specifically listed. Acts as a "catch-all" fallback.
Training crawlers — Bulk-collect content to train future AI models. Your content becomes part of model weights but is not directly attributed. High server load, indirect long-term value. robots.txt is your control surface for opting out. Consider rate-limiting.
Search Index crawlers — Proactively crawl your site to build a searchable index, like Googlebot does for Google Search. When users later ask questions, the AI searches this pre-built index. robots.txt is your control surface for search visibility. Prioritize allowing these.
User Fetcher (RAG) crawlers — Fire only when a human user asks a question, fetching your page in real-time to augment that specific response. This is Retrieval-Augmented Generation. robots.txt compliance varies by company — some honor it (Claude-User), some don't (ChatGPT-User). Allow, but don't rely on robots.txt for blocking.
Ad Validation crawlers — Visit landing pages submitted as ads to verify safety and policy compliance before the ad is allowed to serve. Currently OpenAI's OAI-AdsBot is the only major example (added April 2026). Blocking these does not protect content but does break your ability to advertise on the AI platform. Allow — especially if running AI-platform ad campaigns.
Content-Signal — Cloudflare's proposed extension to robots.txt (Sept 2025) that adds a second layer: if the crawler is allowed to fetch the page, what may it do with the content afterward? Three signals:
search (index for results), ai-input (real-time AI inference — the GEO citation surface), ai-train (model training). Voluntary, unratified, but deployed across 3.8M+ Cloudflare-managed domains by default. Configure in Section 5 below.
1. Select Your Strategy
🚀
Maximize Visibility
Allow all crawlers for maximum AI citations
⚖️
Balanced
Allow search index + RAG, rate-limit training
🛡️
Conservative
Search index + RAG only, block training
2. Configure AI Crawlers
Crawler type legend
Training
Bulk model training — indirect long-term value
Search Index
Proactive indexing for AI search — controllable via robots.txt
User Fetcher (RAG)
Live retrieval at query time — robots.txt may not apply
Validates ad landing pages — blocking may break ad serving
3. Paths to Block
Pages that should NOT be crawled by AI systems (applied to all non-blocked crawlers)
Comma-separated paths. Include trailing slash.
5. Content Signals Policy
Unratified Proposal
Voluntary Compliance
Cloudflare-Deployed
About Content Signals Policy
The acquisition vs. usage distinction. Traditional robots.txt directives (
Allow / Disallow) control whether a crawler may fetch your pages. Cloudflare's Content Signals Policy, introduced September 2025 and deployed across 3.8M+ domains, adds a separate layer that controls how the content may be used after it has been fetched. The two layers operate independently: a Disallow-ed page is never retrieved, so its Content-Signal is moot; an Allow-ed page is fetched but the signal binds the crawler's downstream usage. The signals below apply to every allowed and rate-limited crawler. To give a specific bot a different policy, use the Content-Signal control on that bot's row in Section 2 — an overridden bot is written into its own user-agent group with its own signal line, which (per RFC 9309) it reads in place of the global one.
search
Permission to index your content for search results and excerpts. Setting
no removes you from search citation surfaces.ai-input
Permission to use your content as real-time input for AI answers (RAG, AI Overviews, ChatGPT browsing). Most consequential signal for GEO.
ai-train
Permission to use your content to train or fine-tune AI models. Affects long-term foundational authority, not direct citations.
⚠️ ai-input=no removes you from generative AI citation
Setting
ai-input=no signals to compliant crawlers that your content should not be used in real-time AI answers — including AI Overviews, ChatGPT, Perplexity, and similar surfaces. This is the opposite of typical GEO objectives, which aim to maximize citation in those surfaces. Use this only for deliberate licensing, content protection, or regulatory strategies — not as a default precaution.
Human-readable explanation of the policy
⚠️ Compliance Note
Robots.txt is voluntary. Legitimate crawlers respect it, but some AI systems bypass it by design.
Named exceptions: ChatGPT-User doesn't respect robots.txt for OpenAI's older user-initiated browsing pathway (removed December 2025); OpenAI's newer ChatGPT Agent shares the same UA token but does respect it. Perplexity-User ignores robots.txt per Perplexity's own docs. Google-Agent (Project Mariner) ignores robots.txt entirely — Google classifies it as a user-triggered fetcher. Agentic AI browsers (ChatGPT Atlas, Perplexity Comet) use standard Chrome user-agents with no distinguishing token.
Emerging fix — Web Bot Authentication. An industry pattern (grounded in RFC 9421 HTTP Message Signatures) is emerging: AI crawlers cryptographically sign their requests, receiving servers verify the signatures. ChatGPT Agent already signs every outbound request. Enforcement is happening at the CDN/WAF layer (Cloudflare's AI Crawler Verification System launched June 2026; Fastly and Akamai researching parallel implementations). Why this matters: Cloudflare's May 2026 Radar report estimated 31% of traffic identifying as GPTBot was actually spoofers — so an Allow rule for a spoofed identity is meaningless without verification.
What to do: For crisis or sensitive content, assume that once a page is publicly accessible, AI assistants may fetch and quote it immediately — regardless of what your robots.txt says.
Emerging fix — Web Bot Authentication. An industry pattern (grounded in RFC 9421 HTTP Message Signatures) is emerging: AI crawlers cryptographically sign their requests, receiving servers verify the signatures. ChatGPT Agent already signs every outbound request. Enforcement is happening at the CDN/WAF layer (Cloudflare's AI Crawler Verification System launched June 2026; Fastly and Akamai researching parallel implementations). Why this matters: Cloudflare's May 2026 Radar report estimated 31% of traffic identifying as GPTBot was actually spoofers — so an Allow rule for a spoofed identity is meaningless without verification.
What to do: For crisis or sensitive content, assume that once a page is publicly accessible, AI assistants may fetch and quote it immediately — regardless of what your robots.txt says.
Content Signals are preferences, not enforcement. Widely deployed but not yet a ratified standard.
Universal caveat: Google has publicly declined to commit to honoring Content Signals for AI Overviews.
IETF status: The IETF AIPREF Working Group's August 2026 milestone has slipped. The vocabulary draft (draft-ietf-aipref-vocab-06) is on the Proposed Standard track but notes it "does not yet have full consensus." Current IETF term is
Off Cloudflare specifically: As of August 2026 there is no published evidence that any major AI operator independently honors Content-Signal directives when crawling a non-Cloudflare origin. Verified endorsers of the syntax (Reddit, Yahoo, Quora, O'Reilly, AP, Stack Overflow) are Cloudflare customers or separately-adopted RSL supporters. If you're not on Cloudflare, treat the directive as a signaling gesture rather than enforcement, and revisit as IETF AIPREF finalizes.
IETF status: The IETF AIPREF Working Group's August 2026 milestone has slipped. The vocabulary draft (draft-ietf-aipref-vocab-06) is on the Proposed Standard track but notes it "does not yet have full consensus." Current IETF term is
train-ai (not ai-train); the RAG category is still being drafted. The eventual specification may differ syntactically from Cloudflare's current Content-Signal.
Off Cloudflare specifically: As of August 2026 there is no published evidence that any major AI operator independently honors Content-Signal directives when crawling a non-Cloudflare origin. Verified endorsers of the syntax (Reddit, Yahoo, Quora, O'Reilly, AP, Stack Overflow) are Cloudflare customers or separately-adopted RSL supporters. If you're not on Cloudflare, treat the directive as a signaling gesture rather than enforcement, and revisit as IETF AIPREF finalizes.
If you use Cloudflare: A default-block policy for Training and Agent crawlers takes effect September 15, 2026.
What: Starting September 15, 2026, Cloudflare will block Training and Agent crawlers by default on ad-supported pages. Only Search crawlers stay allowed by default. Mixed-use crawlers that combine functions get blocked entirely. Applies automatically to all new Cloudflare customers, all new sites added by existing customers, and all existing free-tier customers.
Impact: Your robots.txt Allow rules may be silently overridden by Cloudflare's WAF-layer defaults after this date. If your file has
What to do: If your site is behind Cloudflare, audit your Cloudflare Security settings before September 15. Paying customers can override the defaults from the dashboard; free-tier customers need to explicitly opt out to allow AI crawlers. Source: Cloudflare "Content Independence Day" announcement, July 1, 2026.
Impact: Your robots.txt Allow rules may be silently overridden by Cloudflare's WAF-layer defaults after this date. If your file has
Allow: GPTBot but you haven't opted out in Cloudflare's dashboard, GPTBot will still be blocked at the edge.
What to do: If your site is behind Cloudflare, audit your Cloudflare Security settings before September 15. Paying customers can override the defaults from the dashboard; free-tier customers need to explicitly opt out to allow AI crawlers. Source: Cloudflare "Content Independence Day" announcement, July 1, 2026.
The
llms.txt proposal is intentionally excluded from this generator — no major AI crawler reads or honors it as of August 2026.