Robots.txt Generator
Generate robots.txt rules for user agents, allowed paths, blocked paths, and a sitemap URL.
About this tool
What is Robots.txt Generator?
Create a basic robots.txt file for a website by specifying user agents, allowed paths, disallowed paths, and an optional XML sitemap location. The generated text can be reviewed and copied before publishing it at the root of a website.
How to use
Get your result in a few steps
- 1
Enter paths you want crawlers to disallow.
- 2
Enter any explicitly allowed paths.
- 3
Choose a user agent or use * for all crawlers.
- 4
Optionally provide a sitemap URL, then generate and copy the robots.txt file.
Frequently asked questions
Questions about Robots.txt Generator
Does robots.txt prevent a URL from appearing in search results?
Not necessarily. Robots.txt primarily controls crawling. A URL can sometimes still be discovered or indexed through links even when crawling is blocked.
Where should robots.txt be placed?
It is normally served from the root of a host, such as https://example.com/robots.txt.
Can a robots.txt mistake affect crawling?
Yes. Incorrect rules can block important sections of a site, so generated rules should be reviewed before publishing.
Keep exploring
Related tools
FAQ Schema Generator
Generate FAQPage JSON-LD from questions and answers.
Heading Structure Analyzer
Inspect H1-H6 headings in HTML and identify basic hierarchy issues.
Hreflang Generator
Generate alternate-language hreflang link tags for international pages.
Keyword Density Checker
Count keyword or phrase occurrences and calculate their density within a block of text.
Meta Tag Generator
Generate title, description, canonical, robots, and author HTML meta tags for a web page.
Open Graph Preview
Generate Open Graph tags and preview how page information may look when shared.