robots.txt in 2026: Rules, Mistakes, and Deprecated Directives

robots.txt is the oldest and simultaneously the most underestimated file in the SEO toolkit. A single wrong Disallow can cut a site off from indexation for months, and a single missed Clean-param can multiply duplicate pages in Yandex's index for years. In this article we break down what should and should not be in robots.txt as of 2026.

What is robots.txt — briefly

It is a plain-text file located at the root of your site at https://yourdomain.com/robots.txt. Before crawling a site, a search engine robot downloads this file and reads its instructions — which pages may be indexed and which may not. This is a strong recommendation (not a hard directive), but both Yandex and Google respect it.

robots.txt operates at the URL-pattern level, not at the individual page level. If you want to hide a specific page from search results, use the meta robots noindex tag or the X-Robots-Tag HTTP header — those are more reliable.

The 2026 baseline template

The minimal valid robots.txt for a commercial site in 2026 looks like this:

User-agent: *
Allow: /

User-agent: Yandex
Allow: /
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&yclid&gclid&fbclid

Sitemap: https://yourdomain.com/sitemap.xml

That's it. No comments, no deprecated directives, no redundant blocks. Below we explain why.

What should no longer be there

Host: is deprecated

Before 2018, Yandex used the Host: directive to specify the canonical version of a domain (with or without www). In spring 2018, Yandex dropped support for it — switching to standard 301 redirects and <link rel="canonical"> in HTML. Google never understood Host: at all.

If your robots.txt still contains Host: yourdomain.com — remove it without regret. It is simply noise that nobody reads.

Crawl-delay for Google

The Crawl-delay: N directive tells a search engine to wait N seconds between requests. Google never officially supported it and confirmed in 2019 that it simply ignores it. Googlebot's crawl rate is configured in Google Search Console (Settings → Crawl rate), not in robots.txt.

In Yandex, Crawl-delay worked until recently, but it was phased out in 2024–2025. If your server struggles under the load from bots, that is a server problem — not a reason to impose artificial delays.

Comments in robots.txt

You often see lines like # Block indexation of /admin/ in these files. This is fine in scripts and code, but in robots.txt it is unnecessary noise. The file is intended for robots, not humans. Comments do not function as directives, they consume crawl budget when the file is downloaded, and they provide no benefit whatsoever.

If you need to document why robots.txt is configured a certain way, keep that in internal documentation and leave the file itself clean.

Disallow for non-existent paths

Do you like listing every possible service path "just in case," even if those paths don't actually exist on the site? This is a classic mistake copied from templates:

Disallow: /admin/
Disallow: /backup/
Disallow: /tmp/
Disallow: /old/
Disallow: /api/
Disallow: /search-index.json

If these paths don't exist, there is no need to block them. Every extra line gives the crawler a reason to try visiting those paths, only to receive a 404. A clean file with real restrictions is better than a cluttered "protective" list.

What Yandex needs separately

Clean-param — mandatory

This is the directive that 90% of sites ignore, which is why they end up drowning in duplicates. Clean-param tells Yandex: "these GET parameters do not change the page content — don't treat pages with different parameter values as duplicates."

What you must add:

Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content
Clean-param: yclid&gclid&fbclid&_openstat
Clean-param: msclkid&dclid

You can combine them into one line using &. If your site has its own technical parameters (such as ?ref=, ?source=, ?affiliate=) — add those as well.

Without Clean-param, a site running a Yandex.Direct advertising campaign can accumulate thousands of duplicates in the index within a month: the same page repeated with different UTM tags. The search engine tries to differentiate them, your content ends up competing with itself, and rankings suffer.

Sitemap link

You can specify the full path to your sitemap.xml in robots.txt: Sitemap: https://yourdomain.com/sitemap.xml. This signals to search engines that a sitemap exists and where to find it.

Google and Yandex will read the sitemap without this directive too (if it is in the standard location), but including it does not hurt. One important detail: the link must be absolute with a protocol, not relative.

Common mistakes that break SEO

Disallow: / in production

The most catastrophic mistake. It typically appears after migrating from a staging server:

User-agent: *
Disallow: /

This means "do not index anything at all." The site drops out of Yandex and Google search results within a week. Recovery requires not just fixing the file but waiting several weeks for search engines to re-crawl the site.

Check robots.txt immediately after every deploy. This is the second most critical check after "does the site load."

Blocking CSS/JS indexation

In the old days, SEO specialists sometimes added Disallow: /css/ or Disallow: /js/ based on templates from 2012. Today this causes harm: both Google and Yandex render pages like a browser — they need to load CSS and JS to see the final content.

If a search engine cannot access your styles and scripts, it may decide your site is "broken" and demote its rankings. Never block /static/, /_next/, /wp-content/, /assets/, or similar directories.

Blocking image indexation

A similar story: Disallow: /images/ or Disallow: /uploads/ seemed harmless. But this shuts out Google Images and Yandex.Images — a separate source of traffic. For e-commerce stores and media sites this is a serious loss.

If you do not need images indexed, it is better to use X-Robots-Tag: noindex in HTTP headers for specific file types via an nginx rule, rather than blocking an entire directory.

Trailing slash in Disallow

Disallow: /admin and Disallow: /admin/ are different rules. The first blocks the page /admin AND /admin/login AND /administrator/. The second blocks only paths that start with /admin/.

If you want to block a specific subsection, add the trailing slash. If you want to block a prefix, omit it.

Robots.txt for different platforms

A standard robots.txt works for most sites. But there are nuances.

WordPress usually generates its robots.txt dynamically. Do not place a physical file — that creates a conflict. Configure it through the Yoast SEO or RankMath plugins.

Bitrix in newer versions has robots.txt settings in the admin panel. Editing the physical file is pointless — Bitrix will overwrite it on the next update.

Next.js (as we use) — use robots.ts in the app/ directory, where you export a MetadataRoute.Robots object. This is the TypeScript-first approach, more convenient than a physical file.

Astro — a static file in public/robots.txt, no alternatives.

How to check that everything works

After deploying:

Open https://yourdomain.com/robots.txt in a browser — it should display your file with the correct Content-Type: text/plain
Google Search Console → URL Inspection → enter any URL from your site → should show "URL is on Google" (if the site is already indexed) and "Crawling: Allowed"
Yandex Webmaster → Tools → robots.txt Analysis → enter the URL and click "Check" — shows which rules are applied
Bing Webmaster Tools → robots.txt Tester (same principle) — especially important for ChatGPT citation visibility

Summary

A clean, minimal robots.txt is the best robots.txt. No deprecated directives (Host, Crawl-delay), no comments, no blocks for non-existent paths. Mandatory Clean-param for Yandex, mandatory sitemap link. Never block CSS/JS.

If your robots.txt raises any doubts — send it to us and we will check it in 30 minutes. Free of charge. Additional resources on the topic are in our SEO services.