Checker Settings
Example Data Table
| URL | HTTP Status | X-Robots-Tag | Meta Robots | Overall Noindex | Notes |
|---|---|---|---|---|---|
| https://example.com/private | 200 | noindex, nofollow | (none) | Yes | Header blocks indexing without editing HTML. |
| https://example.com/staging/page | 200 | (not present) | robots: noindex | Yes | Meta tag prevents indexing for all bots. |
| https://example.com/blog/post | 200 | (not present) | robots: index, follow | No | Indexable when no other blocking signals exist. |
Examples are illustrative; run checks on your own URLs for real signals.
Formula Used
This checker uses a practical SEO decision rule:
- Noindex detected if X-Robots-Tag contains noindex, OR any robots-related meta tag contains noindex.
- Indexable when HTTP status is 200–399 and no noindex directive is found.
- Unknown when the page body is not fetched or the response is an HTTP error.
Note: This tool checks noindex directives. It does not fully evaluate robots.txt, authentication, JavaScript rendering, or crawl budget constraints.
How to Use This Calculator
- Enter a page URL, then select a user-agent if needed.
- Enable body fetching to detect meta robots directives.
- Turn on redirects if your URL forwards to another page.
- Click Check Noindex to view the summary above the form.
- Download CSV or PDF to archive the audit result.
Article
HTTP and crawl outcomes
A 200 status often signals a renderable page, but indexing still depends on directives. In audits, track the share of 200–399 responses versus 4xx/5xx. If errors exceed 2–5% across templates, fix routing and canonicalization before tuning robots signals. Also monitor median response time; when it rises above 1.0–1.5 seconds, crawlers may sample fewer URLs.
X-Robots-Tag coverage
Server headers can enforce rules on HTML, PDFs, images, and gated pages. Many teams apply noindex at the edge for staging or customer portals. Measure presence rate and directive mix; a sudden rise in “noindex, nofollow” after a deploy is a common incident signature. Treat “nosnippet” and “noarchive” as separate flags because they change SERP presentation.
Meta robots consistency
Meta robots is easiest to change in templates, yet it is frequently duplicated or overridden by user‑agent specific tags. Count how many robots, googlebot, and other agent tags exist per page. More than one robots meta tag is a red flag for conflicting CMS modules. A reliable pattern is one global robots tag plus optional agent tags only when you intentionally diverge.
Redirect chain impact
Redirects change which page is evaluated by search engines. Monitor average redirect hops; keep it at 0–1 for critical pages. When following redirects, ensure the final destination does not inherit a noindex header from an intermediate host or CDN rule. If a page flips between 301 and 302, log dates and stabilize the redirect type to protect signals.
Canonical alignment
Canonical links do not cancel noindex, but they guide consolidation. Compare canonical URLs to final URLs and watch for cross‑domain canonicals that map to noindexed pages. A clean setup uses self‑referencing canonicals for indexable pages and stable preferred versions. If canonicals point to parameterized URLs, quantify how many variants exist and consolidate to a clean target.
Reporting and remediation
Export results to share with developers and content teams. Prioritize pages where status is 200, noindex is “Yes,” and the page is meant to rank. Then separate fixes into header rules, template meta, and deployment configuration to reduce reoccurrence. Recheck after changes and aim for a weekly noindex drift rate near zero across key sections and templates.
FAQs
1) What does “noindex” mean for a page?
Noindex is a directive that asks search engines not to include a page in their index. It can be sent via meta robots tags or the X-Robots-Tag HTTP header.
2) Which signal is stronger: meta robots or X-Robots-Tag?
Both are valid. Headers are convenient for non-HTML files and edge rules, while meta tags are common for HTML templates. If either contains noindex, treat the page as non-indexable.
3) Why should I enable “Follow redirects”?
Search engines evaluate the final destination. Following redirects shows whether the last URL carries a noindex header or meta directive, preventing false assumptions based on the starting URL.
4) Does canonical override noindex?
No. Canonical suggests the preferred URL for consolidation, but noindex blocks indexing. Use canonical to guide selection among indexable pages, not to bypass a noindex directive.
5) Why can results differ by user-agent?
Some sites serve different headers or HTML to bots versus browsers. Testing with common crawler agents helps detect conditional rules, such as noindex for staging, paywalls, or geo-specific experiences.
6) What should I fix first when “Noindex: Yes” appears?
Confirm the page should rank, then identify the source: header rule, template meta, or CDN configuration. Remove or narrow the directive, redeploy, and recheck to ensure the signal is gone.