Robots.txt Generator

Build a correct robots.txt file to guide search engine crawlers, link your sitemap and optionally block AI training crawlers.

FreeNo sign-upRuns in your browserMobile-friendly

Loading tool…

Need help with this tool or found a problem? WhatsApp support Email

What does the Robots.txt Generator do?

robots.txt is a plain-text file at the root of your site (for example https://example.com/robots.txt) that tells crawlers which paths they may fetch. This generator writes valid rules, adds your sitemap location and can add separate blocks for AI and dataset crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended.

Important: robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of search results, use a noindex meta tag instead (and do not block it in robots.txt, or crawlers cannot see the tag).

How to use the Robots.txt Generator

  1. Choose whether to allow or block all crawlers by default.
  2. List any paths to disallow, such as /admin/ or /cart/.
  3. Add your sitemap URL.
  4. Optionally tick AI crawlers to block.
  5. Download robots.txt and upload it to your site root.

Examples

Typical site

User-agent: * / Disallow: /admin/ / Sitemap: https://example.com/sitemap.xml

Common use cases

  • Setting up a new website
  • Blocking admin, cart and search-result pages from crawling
  • Opting out of AI training crawlers
  • Blocking a staging site entirely

Features and benefits

  • Private by design — your data is processed on your device and never uploaded.
  • Free with no account, no watermark and no usage limits.
  • Works on phones, tablets and desktops in any modern browser.

Limitations to know

  • Well-behaved crawlers follow robots.txt, but it is not a security measure — never rely on it to hide private content.
  • Google ignores Crawl-delay.

Helpful guides

Frequently asked questions

Where do I upload robots.txt?

To the root of your domain so it is reachable at https://yourdomain.com/robots.txt.

Does blocking a page remove it from Google?

No. Use a noindex meta tag or remove the page. robots.txt only stops crawling.

Will blocking GPTBot affect Google search?

No. GPTBot is OpenAI’s crawler. Google-Extended controls Gemini training and does not affect Google Search ranking.

Embed this tool on your website

Add the free Robots.txt Generator to your blog or website. Paste the code into your page’s HTML; the tool runs in your visitors’ browsers and stays up to date automatically. Please keep the credit line below the tool.

Free to use on any website. The tool runs inside an iframe served from viptools.pk; nothing your visitors type is sent to your site or ours. Preview the embedded version.

All seo tools

XML Sitemap Generator

Turn a list of URLs into a valid XML sitemap.

Meta Tag Generator

Generate title, description, robots, viewport and canonical tags.

Canonical URL Generator

Normalise URLs and generate rel=canonical tags.

Redirect Rule Generator & Guide

Generate 301/302 redirect rules for Cloudflare, Netlify, Apache and Nginx.

Hreflang Tag Generator

Generate hreflang link tags for multilingual and multi-regional sites.

SEO Content Outline Generator

Build a structured article outline from a topic and keywords.