Generated robots.txt
Your result appears here while you edit settings below.
User-agent: * Allow: /
Generator settings
Sitemap settings
Crawler groups
Add user-agents and ordered path rules.
AI crawler controls
These are editable starter controls. Verify crawler names with current provider documentation.
Import existing robots.txt
Syntax and SEO checks
Warnings update from the generated output.
URL testing simulator
Common directory presets
Select paths and add them to the first crawler group.
Formula used
The generator builds ordered groups from your selected user-agents. Each group contains Allow and Disallow path rules. Sitemap directives are then appended as absolute URLs. The tester compares matching paths using the longest applicable rule. Allow wins when equal matching rules have equal specificity.
How to use this calculator
- Enter your website URL and choose a default policy.
- Apply a site or CMS preset when useful.
- Add crawler groups and customize their path rules.
- Add sitemap URLs and optional AI crawler controls.
- Review validation warnings before publishing the file.
- Test important URLs against relevant crawler names.
- Download the final file as
robots.txt. - Place it at your website root after review.
Robots.txt examples
| Goal | User-agent | Directive | Example |
|---|---|---|---|
| Allow everything | * | Allow | Allow: / |
| Block everything | * | Disallow | Disallow: / |
| Block a directory | * | Disallow | Disallow: /private/ |
| Allow an exception | * | Allow | Allow: /private/public.html |
| Add a sitemap | — | Sitemap | Sitemap: https://example.com/sitemap.xml |
| Match parameters | * | Disallow | Disallow: /*?sort= |
Robots.txt guidance
Robots.txt gives crawler instructions for a specific website host. It belongs at the root path named robots.txt. Search crawlers fetch this file before requesting many pages. Rules generally start with one or more user-agents. Each group then contains Allow or Disallow path directives. Sitemap lines can point crawlers toward XML sitemap files. These instructions manage crawling rather than secure private information.
A Disallow rule does not create authentication or privacy. Anyone can often read the public robots.txt file directly. Sensitive pages need proper authorization and server-side access controls. Search results may sometimes reference blocked URLs without content. Use supported indexing controls when removal from search matters. Keep valuable CSS and JavaScript resources crawlable when needed.
User-agent groups let different crawlers receive different instructions. The asterisk user-agent acts as the broad fallback group. Specific crawler groups can override that general crawling policy. Path matching becomes important when rules overlap each other. More specific matching paths usually determine the final decision. Equal specificity should prefer an Allow rule over Disallow. Test important URLs before uploading changes to production websites.
Wildcards can simplify patterns across many related website URLs. An asterisk can match changing text within supported patterns. A trailing dollar marker can represent a URL ending. Support details can vary across crawlers and search providers. Crawl-delay also has different support among crawler implementations. Treat nonstandard directives as optional crawler-specific configuration only. Verify provider documentation before relying on specialized crawler behavior.
Sitemap directives should normally use complete absolute website URLs. Multiple Sitemap lines can describe several sitemap files or indexes. Keeping sitemaps current helps crawlers discover important public pages. Validation catches common mistakes before publication causes crawling issues. Duplicate rules add clutter and can make maintenance harder. Accidental sitewide blocking can immediately affect crawler access. Staging websites may intentionally block crawling during development periods.
CMS presets are starting points rather than universal final rules. Plugins and custom routes can change important website directories. Ecommerce sites often contain carts and account-related URL sections. Blogs may need fewer restrictions than complex online stores. Forums can create search paths and duplicate filtered URLs. Documentation websites may expose generated asset directories to crawlers. Review every preset against your actual website architecture.
AI crawler controls deserve the same careful review process. Crawler user-agent names can change as providers update products. A custom field keeps this generator useful after such changes. Blocking a crawler also does not protect confidential information. Private content still requires authentication and appropriate authorization controls. Publish only rules you understand and expect to maintain. Careful rules protect crawling without exposing private website data.
File information
robots.txt
UTF-8
text/plain