Does Disallow remove a page from Google?

No. Disallow blocks crawling, but a disallowed URL can still appear in results if other sites link to it. To remove a page from the index, allow crawling and add a noindex meta tag, or use the removals tool.

Should I add a crawl-delay?

Usually not. Googlebot ignores crawl-delay; some other crawlers honour it. Only add one if a specific bot is overloading your server.

SEO Tool

Free Robots.txt Generator

Generate a robots.txt file with sensible defaults — allow crawling, block admin and private paths, set your sitemap URL, and an optional crawl-delay.

Send Us a Message

Website URL

Used to build the Sitemap line.

Crawl-delay (optional)

Seconds. Googlebot ignores this; some crawlers honour it.

Additional disallowed paths

One per line, e.g. /cart/ or /thank-you/.

robots.txt preview

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Place this file at your site root (example.com/robots.txt). robots.txt controls crawling, not indexing — a disallowed URL can still be indexed if linked elsewhere, so use a noindex meta tag to truly exclude a page.

Loading tool…

How to use

1Enter your website URL so the tool can build your sitemap line.
2Choose to allow all crawling and toggle blocking of admin/private paths.
3Add any extra disallowed paths and an optional crawl-delay.
4Copy the output or download robots.txt and place it at your site root.

What this means

robots.txt tells crawlers which paths they may request. It controls crawling, not indexing — a blocked page can still be indexed if linked elsewhere, so use noindex meta tags for true exclusion. Place the file at example.com/robots.txt.

Want this done for you?

AI SEO & Content

AI SEO and content automation — keyword clustering, content briefs, blog production, internal linking, schema, and refresh workflows that grow organic and AI-search visibility.

Explore

AI Website Design

Conversion-focused websites and landing pages built with AI chatbot setup, WhatsApp lead capture, contact forms, basic SEO, and automation-ready structure.

Explore

Frequently asked questions

At the root of your domain, e.g. https://example.com/robots.txt. Crawlers always look for it there; subfolder locations are ignored.

Turn these tools into a growth system

Free tools are a great start. Get a free AI Growth Audit and we'll map the AI marketing, agents, and automation that move your business fastest.