DEVELOPER & SEO TOOLS

Robots.txt Generator 🤖

Create a robots.txt file with crawler rules, sitemap URLs and optional crawl delay settings.

Crawler directivesAllow & disallow rulesCopy or download

Crawler Rules

Website URL

User-agent

Crawl delay

Custom delay

Disallow paths

Allow paths

Sitemap URLs

Add https:// when missing Auto add /sitemap.xml Include helpful comments Clean and sort paths

🛡️ This robots.txt generator runs locally in your browser. Your rules are not uploaded by this tool.

Generated Robots.txt

Rules generated

Enter rules to generate a robots.txt file.

Disallow rules

Allow rules

Sitemaps

Status

Waiting

Upload location:

https://example.com/robots.txt

File name:

robots.txt

Robots.txt output

Upload this file to your website root, for example https://example.com/robots.txt.

Robots.txt Report

Robots.txt checks will appear here.

User agent *

Crawl delay None

Website root Missing

Crawling mode Custom

Check	Result
Current status	Waiting for input

Robots.txt Generator FAQ

How do I create a robots.txt file?

Add your website URL, choose a user agent, enter paths you want to allow or disallow and add your sitemap URL. Then copy or download the generated robots.txt file.

Where do I upload robots.txt?

Upload the file to the root of your website. For example, if your site is example.com, the file should be available at https://example.com/robots.txt.

What does User-agent: * mean?

User-agent: * means the rules apply to all crawlers that read robots.txt. You can also create rules for a specific crawler, such as Googlebot.

What does Disallow: / mean?

Disallow: / asks crawlers not to crawl the entire site. Use this carefully because it can stop search engines from crawling your pages.

Can robots.txt protect private pages?

No. Robots.txt is not password protection. Sensitive files and private pages should be protected with proper access control, not only crawler instructions.