Robots.txt Generator

Control how Google, Bing, and AI crawlers index your website. Generate standard directives, protect sensitive directories, and link your XML sitemap.

Crawler Directives

XML Sitemap URL (Recommended)

Helps search engine bots discover all your published pages

Crawl-Delay in Seconds (Optional)

Slows down bots on shared servers with limited CPU

Disallow Paths (Hide from Search Engines)

Explicit Allow Paths (Exceptions)

Block AI Training Bots

Adds disallow directives for GPTBot, CCBot, Google-Extended, and Anthropic crawler.

UTF-8 Plain Text

# robots.txt generated by CodeCrest Studio Tools
# https://codecreststudio.com/tools/robots-txt-generator

User-agent: *
Allow: /

Save this file to the root of your web server (e.g. https://yourdomain.com/robots.txt).