X-Robots-Tag: Advanced Indexing Control via HTTP

X-Robots-Tag is an HTTP header that serves the same purpose as meta robots, but at the server level. It is less well-known, yet critically important for controlling the indexing of files where meta tags simply cannot be applied — PDFs, images, videos, CSV files, and any other non-HTML static assets.

This article covers practical use cases for X-Robots-Tag along with nginx configurations for each scenario.

Why X-Robots-Tag Exists

The standard <meta name="robots" content="noindex"> lives inside an HTML document. If the resource is not HTML — a PDF, PNG, MP4, or JSON file — you have no place to put a meta tag. Yet search engines do index such resources: Google crawls your PDF and can surface it in search results as a PDF document.

X-Robots-Tag solves this by operating at the HTTP header level. When the server delivers a file, it appends:

X-Robots-Tag: noindex, nofollow

The search engine crawler reads this header before it even looks at the file content. It processes the directive and does not index the resource.

Supported Directives

X-Robots-Tag accepts all the same directives as meta robots:

• noindex — do not add to the index
• nofollow — do not follow links within the resource
• noarchive — do not store a cached copy
• nosnippet — do not display a snippet in search results
• noimageindex — do not index images on the page
• unavailable_after: [date] — stop indexing after the specified date
• max-snippet:[N] — limit snippet length
• max-image-preview:none/standard/large — control image preview display
• max-video-preview:[N] — control video preview length

Directives can be combined with commas.

Use Cases

Scenario 1: PDF Documents Appearing in Search Results

A common problem. Your site has downloadable PDFs — price lists, specifications, legal documents. Google crawls them and surfaces them in search results. Users land on a raw PDF, bypassing your website entirely.

The nginx fix:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

If you want to keep PDFs in the index but without snippets or image previews:

location ~* \.pdf$ {
    add_header X-Robots-Tag "nosnippet, noarchive, max-image-preview:none" always;
}

Scenario 2: Technical JSON / XML Files

Many sites serve sitemap.xml, feed.xml, manifest.json, and similar files. These should not appear in search results — they are meaningless to an end user.

location ~* \.(json|xml)$ {
    add_header X-Robots-Tag "noindex" always;
}

One important caveat: sitemap.xml is exactly what search engines need to discover your pages. Blocking it from appearing in search results is fine. Blocking crawlers from accessing it is not.

Scenario 3: Thumbnail Images and Previews

In e-commerce, a product card contains a thumbnail while a full-resolution photo lives on a dedicated page. You do not want the thumbnail showing up in Google Images instead of the full photo.

location ~* /(?:thumbnails|previews)/.*\.(jpg|webp|png)$ {
    add_header X-Robots-Tag "noimageindex" always;
}

Scenario 4: Content With an Expiry Date

A promotional page for a sale that ends at the end of the month. Once the promotion is over, the content becomes meaningless — but you cannot delete the page because it ranks in search and has inbound links.

location = /promo/halloween-2026.html {
    add_header X-Robots-Tag "unavailable_after: 2026-11-01" always;
}

After November 1, 2026, the page will automatically drop out of the index. You can later update the content and remove the header — Google will re-index it.

Scenario 5: Test and Staging Domains

A staging site at test.bcorrections.com should never end up in the index. The straightforward approach:

server {
    server_name test.bcorrections.com;
    add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
    # ...
}

This header applies to every resource served from the staging domain — HTML, images, JS, CSS. It guarantees nothing will get indexed.

This is more reliable than a robots.txt with Disallow: /. A robots.txt disallow blocks crawling, but if any external links point to the staging site, Google may still attempt to index it. An X-Robots-Tag at the HTTP level is a stricter directive.

Scenario 6: Large Legacy Archive Sections

An old blog you do not want to delete, but also do not want to waste crawl budget on.

location ^~ /archive/ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

noindex, nofollow means "do not index these pages and do not follow links from them." The search engine will gradually stop spending crawl budget on this section.

Comparison With meta robots

When to use what	meta robots	X-Robots-Tag
HTML page, single rule	✅ Less work	Possible, but overkill
Control by file type	❌ Not possible	✅ Primary use case
Control by URL pattern	❌ Not possible	✅ Via regex in nginx
Managing a large number of pages	❌ Must edit each one	✅ One nginx block
Temporary rule (test server)	Requires a deploy	✅ Server-level config

The common approach is to combine both: HTML pages use meta tags, while files and pattern-based rules use X-Robots-Tag.

Pitfalls

add_header in nginx Drops Inheritance

If a server block contains an add_header directive and a location block also has add_header, the headers from the server block will not apply to that location. You need to repeat them explicitly.

The solution is to use the always keyword:

add_header X-Robots-Tag "noindex" always;

always means "apply this header to all responses, including error responses." It also protects against inheritance surprises.

The Header Does Not Work for Static Assets Served via CDN

If your static files are delivered through a CDN (Cloudflare, BunnyCDN), the headers set in your nginx may never reach the end user — the CDN caches the response and may not pass through your custom headers.

The solution is to configure headers at the CDN level, either through CDN rules (page rules in Cloudflare) or via worker scripts.

The Crawler Must Visit the Resource to See the Header

X-Robots-Tag only takes effect when a search engine actually crawls the resource. If the resource is blocked in robots.txt, the crawler never reaches it and never sees the header. Do not combine a Disallow: rule in robots.txt with X-Robots-Tag — it accomplishes nothing.

Verifying It Works

curl -I https://yourdomain.com/file.pdf | grep -i robots
# Expected output: X-Robots-Tag: noindex, nofollow

You can also use Google Search Console → URL Inspection: enter the resource URL and check how Google sees it. If the header is applied correctly, it will show "Indexing: not allowed."

Summary

X-Robots-Tag is a powerful server-side tool that covers scenarios where meta robots falls short. If your site has PDFs, staging subdomains, archive sections, or large file catalogs — X-Robots-Tag is not optional.

Want an indexing audit of your site with a breakdown of what should and should not be appearing in search results? Get in touch with us.