Free tool · Technical

Robots.txt Generator

Create a valid robots.txt with sitemap reference and optional AI crawler rules.

Site URL

Paths to disallow (one per line)

Don't use robots.txt to hide private content — use authentication or noindex.

Block all crawlers (staging sites only)

Optionally block AI crawlers

A business decision: blocking can keep content out of some AI products, and may reduce visibility in them.

GPTBot — OpenAI (model training)OAI-SearchBot — OpenAI (search)ClaudeBot — Anthropic (model training)PerplexityBot — Perplexity (search index)Google-Extended — Google (Gemini training/grounding control)CCBot — Common Crawl

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Upload to your site root so it's available at /robots.txt.

What robots.txt does (and doesn't)

robots.txt tells well-behaved crawlers which paths not to crawl. It doesn't prevent indexing of URLs linked elsewhere and isn't a security control. Use noindex or authentication for that.

More free tools

Build My Website Stack

Answer 10 questions and get a recommended domain, hosting, platform, AI, SEO, analytics, automation, email and security stack.

AI Website Auditor

Check a URL for titles, meta, headings, canonical, robots, sitemap, structured data, links, alt text and AI-search readiness.

AI Content Optimizer

Analyse content for intent, structure, readability, keywords, internal links, FAQ/schema opportunities and AI-search readability.

Text Anonymizer for AI

Remove emails, phones, card numbers, IDs and names before pasting text into ChatGPT or Claude — then restore them. Runs in your browser.