Indexability Test

What is it?

A page can be hidden from search results in two different ways. A noindex directive, set in the robots meta tag or in an X-Robots-Tag HTTP header, tells search engines to render a page but never include it in their index, which means the page can never appear in search results no matter how strong its content or backlinks. A robots.txt disallow rule works earlier in the process: it tells crawlers not to even request the page, which means they may never see its content, its links, or any noindex tag it carries. Both directives are useful when applied deliberately to thin or utility pages, but devastating when left over from staging or applied to content that should rank. This test checks whether your page is currently excluded from search via either mechanism, and where the directive originates.

Why this test matters

A noindex or disallow rule on the wrong page is one of the most common ways important content silently disappears from search results. Staging sites are typically noindexed and often disallowed in robots.txt by default, and either directive is sometimes left in place after launch. SEO plugins often noindex archives, tag pages, or paginated lists by default, which is sensible for some sites but harmful for others. A custom theme can apply noindex globally based on a stale configuration, and a server-level robots.txt can quietly block an entire folder. In every case, the symptom is the same: a page that should rank simply does not appear in search results.

Conversely, both tools are useful when applied deliberately. Internal search results, login pages, thin or duplicate content, and pages that exist for utility rather than for ranking all benefit from being excluded from indexes, and crawl budget is often better spent elsewhere on a large site. The right answer is not “no noindex or disallow rules anywhere” but “applied exactly where they belong and nowhere else.”

Common situations this test catches

• Site-wide noindex left over from staging, the most common cause of accidental exclusion.
• Per-page noindex set in the CMS by mistake, where an SEO toggle was disabled on the wrong page.
• X-Robots-Tag header set by server config, applying noindex to entire URL paths.
• Noindex on category or tag pages that should rank, often because an SEO plugin’s defaults were never customized.
• A robots.txt disallow rule blocking an entire folder or URL pattern, sometimes left over from a staging environment or a CMS migration.
• A page that is both noindexed and disallowed, which usually means the noindex tag was never actually seen by search engines and the page has been invisible for longer than expected.

This test reports whether the page is currently blocked from search by either mechanism and where the directive originates.