4xx and 5xx Errors: The SEO Cost of Broken Requests

When a page returns HTTP 404, it is not just "not found." It is a signal to the search engine: "nothing to index here" — plus wasted crawl budget spent fetching it, plus a friction point for the user who arrived via a broken link. If a site has dozens or hundreds of such pages, that is already a systemic problem.

In this article we break down what each error code means, what it costs the site in SEO terms, and how to fix it.

What Each Code Means

4xx — "Client Error"

The crawler came for a page, but something is wrong with the request. The most common ones:

• 404 Not Found — the page does not exist
• 403 Forbidden — the page exists, but access is denied
• 410 Gone — the page existed but has been permanently deleted
• 451 Unavailable for Legal Reasons — blocked by a regulatory decision

5xx — "Server Error"

The server failed to process the request. The most common ones:

• 500 Internal Server Error — something broke internally
• 502 Bad Gateway — the upstream (PHP, Node, etc.) is not responding
• 503 Service Unavailable — temporarily unavailable (maintenance or overload)
• 504 Gateway Timeout — the upstream is taking too long to respond

What the Search Engine Does with Each Code

404:

On the first encounter, Googlebot retries the page after a few days. After 2–3 failed attempts the page is removed from the index. The signal appears in Google Search Console → Coverage → "Not found (404)".

Neighboring pages are not directly affected. But if there are many 404s, Google draws a conclusion about the overall low quality of the site.

410:

Stronger than 404 — it means "deleted permanently." Google deindexes faster (1–2 recrawls instead of 3–5). Use 410 when you have removed a page for good.

403:

If an important page on the site returns 403, Googlebot retries a few times. If it persists, it deindexes. 403 errors often appear accidentally when a bot-protection layer also blocks Googlebot — worth checking.

500:

"The server broke." The crawler comes back in an hour or two. If the situation recurs regularly, Google reduces the crawl rate — meaning Googlebot starts visiting your site less frequently to avoid overloading it.

503 with a Retry-After header:

If maintenance is planned, serve 503 with the header Retry-After: 3600 (come back in 3600 seconds). This is the correct way to say "temporarily unavailable — do not deindex, we will be back soon." Without Retry-After, Google may treat this as a permanent problem.

The SEO Cost of Errors

Breaking down the specific effects:

1. Wasted Crawl Budget

Every Googlebot request to a 404 page is a missed opportunity to crawl a real page. On a site with 1,000+ pages, crawl budget is limited — each 404 means one fewer useful page crawled per day.

On very large sites (e-commerce with 50,000+ SKUs), crawl budget is a critical resource. See the post on Googlebot and crawl budget for more detail.

2. Loss of Trust

If your sitemap contains 5,000 URLs and 30% of them return 404, Google reduces trust in the entire sitemap. Over time it stops crawling even valid URLs as actively.

3. Behavioral Signals

A user arrives from search results, lands on a 404, and bounces back to the SERP within 5 seconds. The search engine sees this — pogo-sticking. For the page the user left, it is a negative signal.

4. External Links Pointing to Nothing

If other sites link to your URL and it returns 404, the link equity is lost. This is especially painful after a site migration where URLs changed but redirects were not set up.

5. 5xx Is More Critical Than 4xx

If Googlebot visits your site and the server responds with 500, that is a signal: "you are not ready for visits." Google remembers and reduces the crawl rate. The behavior is similar to how Google downgrades slow servers in priority — it simply does not want to "break" you with extra requests.

Where to Find Errors

Google Search Console

Indexing → Pages → Why pages aren't indexed. The breakdown there includes:

• Not found (404) — list of URLs Google could not find
• Server error (5xx) — list of URLs where the server failed
• Redirect error — redirect loops or redirects leading to 404

Yandex Webmaster

A similar report under Indexing → Pages in Search → Excluded.

Server Logs

The most honest source. Open access.log for the past month and filter by 4xx and 5xx status codes:

awk '$9 ~ /^[45][0-9]{2}$/' access.log | head -100

This shows all requests with errors. Group by URL to find the most frequent offenders.

Screaming Frog

Run a full site crawl — report in Internal Links → Status Codes. Any internal links pointing to 4xx or 5xx are candidates for fixing.

How to Fix Them

404 — Internal Links

If your site has <a href="/old-page/"> where /old-page/ no longer exists:

1. If a similar page exists → update the link or set up a 301 redirect
2. If the page is not needed → remove the link from all locations
3. If the page is gone for good → return 410 (not 404) — Google deindexes it faster

404 — External Links

Other sites link to you, but your URL has changed. Options:

1. 301 redirect from the old URL to the new one — the best option. Link equity is passed along.
2. 410 Gone — if the page is simply dead and there is no replacement
3. Custom 404 page with navigation — so users are not lost on the site

5xx — Server Errors

5xx means a real problem on the server. It cannot be fixed by "adjusting SEO settings" — you need to dig into the root cause:

1. PHP/Node crashed → check logs (error.log)
2. Database not responding → check load and connection pool
3. Timeout → speed up queries
4. Out of memory → increase RAM

5xx is typically an infrastructure task, not an SEO one. But the SEO specialist should monitor and escalate.

Prevention

1. Regular site crawl audits (monthly via Screaming Frog or Sitebulb)
2. Log monitoring for 4xx/5xx anomalies
3. Server downtime alerting (Yandex.Metrica → Site Availability report, or an external monitoring tool)
4. Redirect map during site migrations — never leave old URLs without a 301
5. Custom 404 page with navigation and search — reduces bounce from users landing on broken links

Summary

Broken requests are a silent drag on SEO. A single 404 will not cause damage, but 200 of them on a site over a month is already a serious signal to search engines and a meaningful loss of crawl budget. Regular monitoring and prompt fixes are hygiene that costs little time and delivers tangible results.

Want an audit of your site for 4xx/5xx errors? Get in touch. Free, delivered within 2 business days.