Robots.txt Generator
Create a clean robots.txt file, control crawler access, add sitemap information, preview the rules, and download the completed file for your website.
Configure Robots.txt
Select a preset or create custom crawling rules.
Add paths that crawlers are allowed to access.
Add private, duplicate, or unnecessary paths that bots should not crawl.
Enter the complete URL of your XML sitemap.
Live Preview
Your generated robots.txt file will appear below.
What Is a Robots.txt File?
A robots.txt file is a plain-text file that gives crawl instructions to search engine bots and other automated crawlers. It can allow or disallow access to specific website paths and can also point crawlers to your XML sitemap.
How to Use the Robots.txt Generator
- Select a preset that matches your website type.
- Choose the crawler or user agent you want to control.
- Add allow and disallow paths, one path per line.
- Select common rules such as blocking admin, login, search, feeds, or checkout pages.
- Add your XML sitemap URL.
- Generate, review, copy, or download the robots.txt file.
- Upload it to the root directory of your domain.
Avoid blocking CSS, JavaScript, images, or pages that search engines need to render and understand your website.
Adding a complete sitemap URL can help search engines discover important pages more efficiently.
Your robots.txt file should normally be accessible at example.com/robots.txt.
Basic Robots.txt Example
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
WordPress Robots.txt Example
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-login.php Disallow: /search/ Disallow: /feed/ Sitemap: https://example.com/sitemap.xml
E-commerce Robots.txt Example
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /?add-to-cart= Disallow: /*?orderby= Disallow: /*?filter_ Sitemap: https://example.com/sitemap.xml
Robots.txt Best Practices
- Use precise paths and review every disallow rule carefully.
- Keep important public pages crawlable.
- Add the complete URL of your XML sitemap.
- Test the file after uploading it.
- Review the file whenever the website structure changes.
- Do not use robots.txt as a security tool.
Robots.txt vs. Noindex
Robots.txt controls whether a crawler may access a URL. A noindex directive tells a supported search engine not to include a page in search results. Blocking a page in robots.txt does not guarantee that the URL will disappear from an index because the crawler may be unable to see the noindex directive on the blocked page.
Common Robots.txt Mistakes
- Using
Disallow: /on a live website by mistake. - Blocking CSS, JavaScript, images, or important content folders.
- Trying to protect private information with robots.txt.
- Using incorrect paths or missing the leading slash.
- Assuming every crawler supports every directive.
- Forgetting to update the sitemap URL after a domain change.
Frequently Asked Questions
What is a robots.txt file?
It is a plain-text file that gives crawl instructions to search engine bots and other automated crawlers.
Where should robots.txt be uploaded?
It should normally be uploaded to the root directory so it is accessible at
https://example.com/robots.txt.
Can robots.txt remove a page from Google?
No. It controls crawling, not guaranteed removal from search results. Use an appropriate noindex directive or supported removal method when indexing must be prevented.
Should I add my sitemap to robots.txt?
Yes, adding the complete sitemap URL is useful because it helps crawlers locate the sitemap.
Does every search engine support crawl-delay?
No. Support varies, and some major search engines may ignore this directive.