Create a correct robots.txt file for your website with sitemap, disallow rules and AI crawler blocking.
What is the Robots.txt Generator?
The robots.txt file tells search engine bots which pages they may crawl. This generator builds a clean file with allow/disallow rules, a sitemap link, an optional crawl delay and an option to block AI training crawlers like GPTBot and CCBot.
How to use the Robots.txt Generator
- Choose the default rule and add your sitemap URL.
- List paths to block and paths to always allow.
- Download robots.txt and upload it to your website root.
Frequently asked questions
Where do I put robots.txt?
In the root folder of your domain, so it opens at yoursite.com/robots.txt.
Does disallow remove pages from Google?
No. It stops crawling, but to remove a page from results use a noindex tag.
This robots.txt generator is 100% free, needs no signup and runs in your browser, so your data stays private.
Last updated: · By NodifyTech