Robots.txt Generator Online
Fill in the form and the generator will produce a correctly formatted robots.txt with User-agent, Disallow, Allow directives and a Sitemap reference.
User-agent: *
Disallow:
Clean-param: utm_source&utm_medium&utm_campaign&utm_content&utm_term&fbclid&gclid&yclid&_ga&_ym_uid&_openstat /
Sitemap: https://example.com/sitemap.xmlFrequently asked questions
What is robots.txt and why does it matter?
Robots.txt is a plain-text file placed at the root of a domain that instructs search engine crawlers which pages they may and may not index. A well-configured robots.txt conserves crawl budget and prevents administrative or duplicate pages from entering the index.
Which pages should always be blocked in robots.txt?
Block: /admin/, /cart/, /checkout/, /thank-you/, faceted-navigation duplicates (/?sort=, /?filter=), internal search results (/search/), user account pages, and PDF files that should not appear in search results.
Where does robots.txt need to be placed?
The file must be located at the exact root of your domain: https://example.com/robots.txt. A file placed in a subdirectory or on a subdomain will not be recognised by crawlers.
Should I include a Sitemap directive in robots.txt?
It is not required, but strongly recommended. The line Sitemap: https://example.com/sitemap.xml helps crawlers discover your sitemap faster, especially on newer domains.
Need a full SEO audit?
Tools help with targeted tasks. For systematic growth — request a free review of your site.