Robots.txt Generator Online — Free Tool
Generate compliant robots.txt directives to instruct search engine crawlers on indexing your website.
Robots.txt Generator
Generate a `robots.txt` file locally (no upload).
Tip: Keep disallow rules small and intentional; blocking everything can hurt discovery.
Privacy note: All processing happens entirely in your browser. No data is sent to our servers.
About this tool
robots.txt tells compliant crawlers which paths they may request. Generate User-agent, Allow, and Disallow rules without memorizing syntax.
robots.txt is not access control—sensitive URLs still need authentication.
Common use cases
- Blocking staging or admin paths from indexing
- Declaring sitemap.xml location for crawlers
- Setting crawl-delay for polite bots (non-Google)
- Documenting crawl policy for new sites
How to use
- List user agents to target (* for all bots).
- Add Allow and Disallow path rules.
- Include Sitemap: URL pointing to your sitemap.
- Copy robots.txt to your site root at /robots.txt.
This page is available at /tools/robots-txt-generator/.
Understanding the result
- Disallow prevents crawling, not necessarily indexing if URLs are linked elsewhere.
- Rules are prefix-based unless wildcards supported by specific bots.
- Sitemap directive helps discovery; it does not replace valid sitemap content.
- Google ignores crawl-delay; other bots may honor it.
Related tools
Meta Tag & SERP Preview
Draft your SEO titles and meta descriptions, generate the HTML tags, and visualize them as Google search results.
Open Graph Preview
Test how your content will appear when shared on social media by generating precise Open Graph meta tags.
Sitemap XML Generator
Construct a standardized sitemap.xml file rapidly by mapping your website URLs to assist search engine indexing.
URL Slug Generator
Convert titles to URL slugs with batch mode, stop-word removal, camelCase/PascalCase variants, and full URL preview.
FAQ
Does Disallow hide pages from Google?
It discourages crawling. Use noindex on pages you do not want indexed.
Where does robots.txt live?
Must be at https://example.com/robots.txt at domain root.
Block all crawlers?
User-agent: * with Disallow: / blocks all paths for compliant bots.
Allow vs Disallow order?
Most specific matching rule wins in Google's interpretation for overlapping rules.