Robots.txt & AI crawler generator
Decide which search engines and AI crawlers can access your site, then upload the file to yoursite.com/robots.txt.
# robots.txt generated with marketingbyraj.com/tools/robots-txt-generator User-agent: * Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /?s= Allow: /wp-admin/admin-ajax.php # All AI crawlers allowed (they follow the rules above) Sitemap: https://example.com/sitemap.xml
How to use it
- 1Choose a preset: allow everything, allow search but block AI training, or block everything.
- 2Fine-tune each AI crawler with the switches.
- 3Add folders to keep private (for example /wp-admin/ or /cart/).
- 4Add your sitemap URL, then download and upload robots.txt to your site root.
Questions
Should I block AI crawlers?+
If you want to be cited by ChatGPT, Claude or Perplexity, allow their search bots. Blocking them can make you invisible in AI answers.
What is Google-Extended?+
A Google control that decides whether your content can be used to train Gemini. It does not affect Google Search rankings.
Does robots.txt keep pages private?+
No. It is a request, not security. Use passwords or noindex for truly private pages.
Run the free website audit: SEO, speed, social and AI visibility in about 60 seconds.
More free tools
See all tools →See how your title and description look in Google and on social media, with length checks.
Open tool →Create title, description, canonical, Open Graph and X (Twitter) tags ready to paste into your <head>.
Open tool →Create Person, Organization, LocalBusiness and FAQ JSON-LD so Google and AI engines understand your brand.
Open tool →Build an llms.txt file that points AI assistants to your most important pages.
Open tool →