Technical SEO audits: reading Search Console and prioritising what to fix

You open Search Console and find thousands of pages not indexed, row after row of unfamiliar status names and a few errors in red. Your agency mentions technical issues, but nobody says which ones matter. Or the opposite: the service page you launched weeks ago still isn't in Google, and you have no idea why.

This guide walks through a technical SEO audit in order: how Google discovers, crawls and indexes a page, what the Search Console reports are telling you, which mistakes with robots.txt, noindex, canonicals, redirects, sitemaps and JavaScript genuinely cause damage, and the order in which to deal with what you find.

Crawling, rendering and indexing are separate stages

Google first discovers a page, learning its address from a link on another page or from a sitemap. Then it crawls it: Googlebot downloads the page from your server. Next comes rendering. According to Google's documentation, pages returning a 200 status code are queued for rendering in a headless Chromium, where JavaScript runs, unless a robots meta tag or header says not to index them. Finally there is indexing: Google analyses the content, picks a canonical version from any duplicates and decides whether to index the page at all.

This distinction is the backbone of any audit. Crawled is not indexed, and indexed is not ranking. Each stage has its own problems and fixes; writing more content for a page that isn't being crawled is usually wasted effort.

Accept one thing from the start: Google doesn't index every page, and it doesn't need to. Search Console Help says so plainly: don't expect every URL to be indexed; make sure your important pages are, and that every page that isn't has a valid reason. The aim of an audit is not to get the grey number to zero, but to have the right pages in the right row.

The Page indexing report: what the statuses mean

The Page indexing report splits the URLs Google knows about into indexed and not indexed, and lists the reasons. Your first question should be: where are the pages that make money, meaning service, category and product pages? Export the example URLs and compare them with your own list of priority pages.

Crawled - currently not indexed

Google crawled the page and chose not to index it. The Help page describes this as common and not always a problem. Our experience is that this row is usually a signal about value: very thin pages, near-identical location pages, filter pages that return no results, auto-generated tag archives.

The fix is not to keep hitting Request indexing in the URL Inspection tool. If the page matters, genuinely strengthen it, merge near-duplicates and link to it from your important pages. If it doesn't, noindex it or remove it. A plumbing firm that built thirty town pages where only the town name changes, for example, will fix both the report and the user experience by folding them into a few area pages with real local information.

Discovered - currently not indexed

Google knows the URL but hasn't crawled it yet; according to the Help page, often because it postponed the crawl to avoid overloading the site. On a small site this may last days. On a large one, thousands of URLs waiting for months suggest a slow server or far more low-value URLs than the site needs. Faceted filter combinations and endless calendar pages are the classic culprits.

Duplicate and canonical statuses

Duplicate without user-selected canonical: the page duplicates another and you haven't declared a canonical, so Google picked one. Alternate page with proper canonical tag: the page correctly points to another as canonical; this is expected and needs no action. Duplicate, Google chose different canonical than user: Google ignored the canonical you declared. If your important pages show up in that last row, your signals are contradicting each other.

Other common statuses

• URL marked ‘noindex’: fine if intentional; urgent if it's on a page that matters.
• Blocked by robots.txt: Google isn't crawling the page, which doesn't mean it won't be indexed.
• Indexed, though blocked by robots.txt: Google indexed the URL based on links alone, without reading its content.
• Page with redirect and Not found (404): normal for old addresses; what matters is that internal links and sitemaps don't point to them.
• Soft 404: the page looks like an error page but returns 200; common on empty categories and out-of-stock products.
• Server error (5xx) and Redirect error: always worth investigating; the first is on the server side, the second means a loop or an overly long chain.

robots.txt and noindex do different jobs

robots.txt controls crawling, not indexing. Google's documentation is explicit: robots.txt is not a mechanism for keeping a web page out of Google. If other sites link to a blocked URL, it can still appear in search results, just without a description.

noindex prevents indexing, but only if Google can crawl the page and see the tag. Using both together is one of the most common mistakes we see: the page is blocked in robots.txt and carries a noindex, so Google never sees the noindex and the URL can stay in the index. To remove a page, add noindex first, lift the robots.txt block, and only put the block back once the page has dropped out, if you still need it.

• Pages you want accessible but out of search, such as thank-you pages and internal search results: noindex.
• Low-value URL spaces that put load on the server, such as endless filter combinations or calendars: robots.txt.
• Content nobody should see, such as staging sites or customer areas: password protection. robots.txt is a public file, and listing private folders there advertises them.
• CSS and JavaScript the page needs to display properly: never block them. If Google can't see the page, it can't understand it.

The robots.txt report under Settings in Search Console shows how Google reads your file. After a site relaunch, leftover Disallow lines from the staging environment are the first thing to look for there.

Canonicals, redirects and sitemaps

A canonical is a hint, not an order

The rel=canonical tag tells Google which URL is the preferred version, but Google can choose differently based on other signals. The URL Inspection tool shows the User-declared canonical next to the Google-selected canonical; if they differ, your signals conflict. Internal links, sitemaps, redirects and hreflang should all point to the same URL.

The canonical mistakes we meet most often: a template bug that makes every page declare the homepage as canonical, paginated category pages all pointing to page one, and http and https, www and non-www, trailing slash and no trailing slash versions getting mixed up.

Redirects and chains

Google treats 301 and 308 redirects as a strong signal that the target should be canonical, and 302 and 307 as a weak one. Use 301 for permanent moves. Googlebot generally follows up to ten redirect hops, but that doesn't make chains harmless: each link slows crawling and adds risk. Loops show up in the report as Redirect error, and the page won't be indexed.

Chains build up over successive relaunches: the 2019 address redirects to the 2022 one, which redirects to the current one. Point old redirects straight at the final URL, and link internally to final URLs, not through redirects.

Sitemaps

A sitemap file can hold up to 50,000 URLs or 50 MB uncompressed; larger sites split them and use a sitemap index file. Google ignores priority and changefreq, and uses lastmod only if it is consistently and verifiably accurate. You can submit a sitemap in the Sitemaps report or add a Sitemap line to robots.txt; submission is a hint, not a promise that Google will crawl everything.

Only include URLs that return 200, are canonical and that you want indexed. A sitemap full of redirects, 404s or noindexed pages sends mixed messages, and it is one of the easiest audit findings to fix.

JavaScript-generated content

Google can run JavaScript, but rendering is queued, some bugs only appear for Googlebot and, as Google's documentation points out, not every bot runs JavaScript. That is why Google still recommends server-side rendering or pre-rendering. On a site built with a modern framework, the safest approach is for important content to arrive in the initial HTML response.

• Links must be real link elements: Google only discovers URLs in a tags with an href attribute. Buttons that work purely through click handlers are invisible to crawling.
• Don't build page addresses on the hash symbol (#); use the History API to give each view a real, crawlable URL.
• Don't put noindex in the initial HTML and remove it with JavaScript later; when Google sees noindex, it may skip rendering altogether.
• In single-page apps, redirect missing content to a URL that returns 404 or add noindex; otherwise every error view becomes a soft 404.

To see your page the way Google does, run Test live URL in the URL Inspection tool and then View crawled page. If the main content, headings and links are missing from the rendered HTML, that's where the problem lies.

Internal links and orphan pages

An orphan page is one that receives no links from anywhere on your site. It might be in the sitemap or used in ad campaigns, but it can't be reached by navigating the site. Even when Google finds such pages, they get no importance passed from the rest of the site and often sit in the Crawled or Discovered rows.

To find them, compare two lists. Crawl the site from the homepage with Screaming Frog or a similar crawler; that gives you the pages reachable through links. Then compare it with the URLs in your sitemap, in Search Console and in Google Analytics 4. Every URL that appears in those sources but not in the crawl is an orphan candidate.

• Keep important pages within a few clicks of the homepage; the deeper a page sits, the less often it tends to be crawled and seen.
• Make anchor text describe the target: use phrases like corporate web design rather than click here.
• Link from blog posts to the relevant service and product pages; most sites never build that bridge between content and commercial pages.
• Use breadcrumbs and related-page sections for pages that aren't in the main menu.

Structured data, HTTPS and mixed content

Structured data tells search engines in machine-readable form what a page is: a product, an event, a business, an article. Google recommends JSON-LD. Check your markup with the Rich Results Test, which shows eligibility for the rich result types Google supports; for general schema.org validation beyond Google's features, use the Schema Markup Validator. After launch, the enhancement reports in Search Console list errors by type.

Two caveats. Markup must match what's visible on the page; marking up review ratings that don't appear on the page falls foul of spam policies. And correct markup doesn't promise a rich result. Google also trims the types it supports from time to time, and in 2025 it removed several lesser-used ones from search results. Check the current list on Search Central before investing in a new type.

On HTTPS, the goal is simple: every version of the site (http, www and non-www) should 301 to a single HTTPS address. Mixed content means an HTTPS page loading an image, script or font over HTTP. Modern browsers either try to upgrade such resources to HTTPS or block them, which can leave broken images, a form that doesn't work or a security warning in the address bar. Hard-coded http addresses stored in an older CMS database are the usual source.

Log file analysis and the URL Inspection tool

Search Console tells you what Google thinks; server logs tell you what it does: which URLs Googlebot requests, how often, which status codes it gets and where its time goes. On an e-commerce site with thousands of pages, log analysis often reveals that Googlebot spends most of its time on filter parameters and old redirects while rarely visiting new product pages.

Not every log entry claiming to be Googlebot is genuine; plenty of bots fake the user agent. Google's recommended check is a reverse DNS lookup on the IP address to confirm the domain is googlebot.com, google.com or googleusercontent.com, followed by a forward DNS lookup to confirm it resolves to the same IP. Google also publishes its crawlers' IP ranges as JSON files. Tools such as Screaming Frog Log File Analyser make filtering and reporting easier.

If you can't get at the server logs, the Crawl stats report under Settings in Search Console gives a rougher but useful picture: requests per day, average response time and status code breakdown. Because Google temporarily slows crawling when it receives 5xx errors and 429 responses, rising response times and server errors show up here early.

The URL Inspection tool tells the story of a single page: Crawl allowed, Page fetch, Indexing allowed, and which canonical Google selected. After a fix you can use Request indexing, but there is a daily quota, and according to Google's documentation requesting the same URL repeatedly won't get it crawled any faster; crawling can take anywhere from a few days to a few weeks. Set up Bing Webmaster Tools as well, since Bing's index is also used by other search services.

Where to start: prioritising the findings

An audit tool will hand you hundreds of warnings, and fixing them all at once means fixing none properly. Rank them with three questions: how much traffic and revenue does this issue cost? How many pages and templates does it affect? How much effort is the fix? A single template bug often rescues thousands of pages with one change, and findings like that go to the top.

1. List your revenue-generating pages and check each one in the Page indexing report and the URL Inspection tool.
2. Review the robots.txt report, noindex use and canonical tags at template level.
3. Crawl the site and list redirect chains, 404s and orphan pages.
4. Trim the sitemap to canonical URLs that return 200, and resubmit it.
5. Check JavaScript-dependent templates with Test live URL.
6. Look at Crawl stats or your server logs to see where crawling time goes.
7. Start Validate fix for each resolved issue and compare results at the next audit.

In AYTE Digital's technical SEO audits we rank findings much like this table, by impact, scope and effort, write them up clearly enough for your developers to act on directly and track with you how the fixes show up in Search Console. We don't promise rankings; we build the foundation Google needs to crawl and understand your site without friction.