Robots.txt Generator
Generate robots.txt files for your website. Control search engine crawling with user-agent rules, allow/disallow paths, crawl-delay, and sitemap references.
Adding your sitemap URL helps search engines discover your pages
User-agent: * Allow: /
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a text file placed in the root directory of a website that tells search engine crawlers which pages or sections they can or cannot access. It follows the Robots Exclusion Protocol and is the first file crawlers check before indexing a site.
Where should I place my robots.txt file?
The robots.txt file must be placed in the root directory of your website, accessible at yourdomain.com/robots.txt. It must be at the root level to be recognized by search engine crawlers. Each subdomain needs its own robots.txt file.
Can robots.txt block pages from appearing in Google?
Robots.txt can prevent crawlers from accessing pages, but it does not guarantee pages will not appear in search results. Google may still index a URL if other pages link to it, showing it without a description. To truly prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header instead.
What is crawl-delay in robots.txt?
Crawl-delay is a directive that tells search engine bots to wait a specified number of seconds between requests. It helps reduce server load from aggressive crawling. Note that Google does not officially support crawl-delay (use Google Search Console instead), but Bing and other crawlers do respect it.
Should I add my sitemap URL to robots.txt?
Yes, adding your sitemap URL to robots.txt is a best practice. It helps search engines discover your sitemap and all the pages listed in it. Add it at the bottom of your robots.txt file using the format: Sitemap: https://yourdomain.com/sitemap.xml