Free Robots.txt Generator
Build a clean robots.txt file: choose a default rule, block folders, add rules for specific crawlers such as AI bots, and include your sitemap. Copy or download the result.
How to use the robots.txt generator
Pick a default rule, list any folders you want crawlers to skip, tick the bots you want to block entirely and add your sitemap URL. Copy the output or download it as robots.txt and upload it to the root of your domain, so it is available at yourdomain.com/robots.txt.
What robots.txt can and cannot do
Robots.txt tells crawlers which URLs they may request. It is not a way to keep a page out of Google: a blocked URL can still be indexed if other pages link to it. Google Search Central recommends a noindex tag or password protection for that. Also avoid blocking CSS and JavaScript files that Google needs to render your pages.
Blocking AI crawlers
Several AI companies publish user-agent names for their crawlers, for example GPTBot (OpenAI), Google-Extended (Google AI training), CCBot (Common Crawl) and ClaudeBot (Anthropic). Blocking them in robots.txt relies on each company honouring the file. Blocking Google-Extended does not affect Google Search crawling.
Robots.txt generator FAQ
Is this robots.txt generator free?
Yes. It is free, needs no sign-up and runs in your browser.
Where do I put robots.txt?
In the root folder of your site so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
Does Disallow stop a page from being indexed?
Not reliably. It stops crawling, but a URL can still be indexed from links. Use a noindex meta tag on a crawlable page to keep it out of search results.
Does Google follow Crawl-delay?
No. Google ignores the Crawl-delay rule. Some other crawlers, such as Bing, may respect it.
Should I add my sitemap to robots.txt?
Yes, it is a simple way to help search engines find your sitemap. You can also submit it in Google Search Console.
Related free tools
Want the links built for you? See our link building service.
Need quality links? Get a quote
Linkvista builds guest posts and backlinks for UK SEO agencies. You approve every domain before we start, with 48-72 hour turnaround.
Get a quoteSee pricing