Free tool
Robots.txt Generator
Make a correct robots.txt in a minute: keep private paths out, point crawlers to your sitemap, and decide which AI crawlers may read your site.
Runs in your browser: nothing you type is sent to us.
Your robots.txt
User-agent: * Disallow: /admin Disallow: /api/ Sitemap: https://yoursite.com/sitemap.xml
Upload it to the root of your site, so it opens at yoursite.com/robots.txt. Crawlers only read it there.
How it works
- 1
Pick a starting point
Allow everything, block AI training, block all AI, or block everything on a staging site.
- 2
Add your paths and sitemap
List the folders to keep crawlers out of and the full address of your sitemap.
- 3
Upload the file
Copy or download it, and put it at the root of your site as robots.txt.
Questions
What is a robots.txt file?
A plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which parts of the site they may visit. Search engines and most AI crawlers read it before anything else. It is a request, not a lock: it does not hide a page from people.
Where do I put robots.txt?
At the top level of your domain, so it opens at https://yoursite.com/robots.txt. A robots.txt in a folder is ignored, and each subdomain (blog.yoursite.com) needs its own.
Should I block AI crawlers like GPTBot?
It depends on what you want. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect pages to train models. Search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch pages to answer people and cite sources. Many founders block training and keep search open, so assistants can still recommend them.
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a page that other sites link to can still appear in results, without a description. To keep a page out of results, let it be crawled and add a noindex robots meta tag, or protect it with a password.
Why add my sitemap to robots.txt?
The Sitemap line tells every crawler where your sitemap is, so they find all your pages, including new ones, without you submitting it to each search engine. Use the full address, starting with https://.
Is anything I type sent to you?
No. The file is made in your browser, and nothing is stored or sent to us.
More free tools
- Website AuditCheck a page for SEO, sharing, speed and technical problems and get a score with fixes.
- Link Preview CheckerSee how a link will look on Facebook, X, LinkedIn, Slack and WhatsApp, and fix what is missing.
- Dofollow Link CheckerSee every link on a page and whether it is dofollow, nofollow, sponsored or UGC, and check that a backlink counts.
- SERP Snippet PreviewPreview your title and meta description as search results show them, and see where they get cut.
- OG Image GeneratorMake a 1200 by 630 share image for your page in your browser, then download it as a PNG.
- OG Image GalleryBrowse share image designs and open any of them in the generator to make it your own.
- UTM Link BuilderBuild tracked links for every launch channel, with presets for Product Hunt, X, Reddit and newsletters.
- llms.txt GeneratorWrite the llms.txt file that tells ChatGPT, Claude and Perplexity what your site is and where its key pages are.
- PPP Pricing CalculatorSet fair prices by country: see a suggested discount and price for 190+ countries.
One email a week
The AI products that launched this week and the top of last week's board. No ads in it, and you can leave with one click.
Launching a SaaS or AI product?
List it on Stackboo for free. It is ranked by votes, not budgets, and reviewed by a person.