What is technical SEO?

Technical SEO is the practice of making a website easy for search engines to crawl, render, index and understand. It covers the infrastructure behind your content: robots.txt, sitemaps, canonical tags, site architecture, internal links, page speed, mobile rendering, HTTPS, structured data and redirects. Good content cannot rank if Google cannot reach it, process it, or decide which version to keep.

The 12-step technical SEO checklist (summary)

  1. Crawlability: robots.txt allows what matters, sitemaps list only index-worthy URLs, and crawl waste is under control.
  2. Indexing: canonicals, noindex and duplicate URLs are consistent, and the Search Console page indexing report makes sense.
  3. Rendering: critical content and links exist in the HTML Google receives, not only after JavaScript runs.
  4. Architecture and internal links: important pages sit within a few clicks of the homepage and nothing valuable is orphaned.
  5. Core Web Vitals: LCP, INP and CLS pass at the 75th percentile of real users.
  6. Mobile: the mobile version carries the same content, links and structured data as desktop.
  7. HTTPS and security: one secure, canonical host, no mixed content, no hacked pages.
  8. Structured data: markup for features Google still supports, matching what is visible on the page.
  9. Redirects and migrations: single-hop 301s, no chains, and a URL map for every move.
  10. International: hreflang is reciprocal and points at canonical URLs, if you serve more than one market.
  11. AI crawlers and AI search: you know which bots you allow and what controls Google actually honours.
  12. Monitoring: a short audit routine you repeat monthly and after every release.

Google can only rank what it can efficiently crawl, render, understand, and trust enough to keep indexed.

How to tell if technical issues are your bottleneck

  • Pages you care about are missing from the index, especially category, service or hub pages.
  • Search Console shows growing counts of "Discovered, currently not indexed" or "Crawled, currently not indexed".
  • Traffic jumps or drops after small site changes, such as a theme update or a new plugin.

Start with the index, not the checklist

Before touching anything, confirm what Google actually has indexed versus what you submitted. Most "technical SEO" projects are really one fix, stopping crawl waste on URLs that were never going to rank, and you can only see that gap from the index data, not from a tool score.

1. Crawlability: robots.txt, sitemaps and crawl waste

Diagram showing how search engines crawl and index a website's pages

Most technical SEO wins come from one idea: reduce waste. Googlebot's time should go on pages that deserve to rank, not internal search results and parameter duplicates.

Robots.txt does one job

Robots.txt controls crawling, not indexing. Google supports a small set of rules in it (user-agent, allow, disallow and sitemap) and ignores directives such as crawl-delay. A blocked URL can still be indexed if other pages link to it, just without its content, so robots.txt is the wrong tool for keeping a page out of results.

text
User-agent: *Disallow: /search/Disallow: /cart/Disallow: /*?sort=Sitemap: https://www.example.co.uk/sitemap.xml

Two common mistakes to check for:

  • Blocking CSS or JavaScript that the page needs to render. Google will not render JavaScript from blocked files, so it may see a broken or empty page.
  • A leftover Disallow: / from a staging site that went live. It sounds obvious, and it still happens after relaunches.

Sitemaps that Google trusts

A sitemap is not a dump of every URL your CMS can output. It is a curated list of pages you want indexed. If it is full of redirects, 404s and noindexed URLs, it stops being a useful signal.

  • Only include canonical, 200-status, indexable URLs.
  • Split large sitemaps by type (products, categories, articles, locations) so the Search Console sitemap report shows you which section has the problem.
  • Keep lastmod honest. Only change it when the content meaningfully changes.

Crawl waste and crawl budget

Crawl budget matters most for large sites, but any site with faceted navigation, sort options, tracking parameters or user-generated pages can produce tens of thousands of crawlable URLs by accident. That slows discovery of the pages you actually care about.

  • Keep internal links pointed at clean, primary URLs, never parameterised versions.
  • Decide which filter combinations deserve their own landing pages and keep the rest out of the crawl path.
  • Check the Crawl Stats report in Search Console (Settings, then Crawl stats) for spikes in requests to URL patterns you did not expect.

2. Indexing: canonicals, noindex and the page indexing report

Indexing is Google deciding whether to keep a page and which version to keep.

Robots.txt vs noindex vs canonical

These three tools do different jobs, and mixing them up causes index chaos:

  • Robots.txt controls whether Google fetches a URL.
  • Noindex (meta robots tag or X-Robots-Tag header) tells Google not to index a page. Google has to be able to crawl the page to see it, so never combine noindex with a robots.txt block on the same URL.
  • Canonical (rel="canonical") suggests which of several similar URLs should represent the group. It is a hint, not a directive, and Google weighs it alongside redirects, internal links and sitemap inclusion.

Use noindex for internal search results and thin utility pages. Canonicalise parameter and tracking variants back to the clean URL, but only when the content really is a duplicate. Never canonicalise to a URL that redirects or returns an error.

Duplicate and parameter URLs

The usual suspects are HTTP and HTTPS versions, www and non-www hosts, trailing slash variants, uppercase paths, session IDs, and filter or sort parameters. Pick one preferred format, 301 everything else to it, and make every internal link, canonical tag and sitemap entry use that exact format. Mixed signals are what make Google choose a different canonical from the one you wanted.

Reading the page indexing report

The Pages report in Search Console lists every reason a known URL is not indexed. The reasons that matter most, and what to do about them:

  • Discovered, currently not indexed: Google knows the URL but has not crawled it yet. Often a sign of weak internal linking or too many low-value URLs competing for crawl. Link to it from relevant, crawled pages.
  • Crawled, currently not indexed: Google fetched it and chose not to index it, usually a quality or duplication judgement. Improve or consolidate the page rather than resubmitting it.
  • Duplicate, Google chose different canonical than user: your signals disagree. Check internal links, redirects and the sitemap all point at the version you declared.
  • Excluded by 'noindex' tag: fine if intentional. If a money page appears here, find the template or plugin setting that added it.
  • Page with redirect and Not found (404): normal in small numbers, unless pages that should be live appear.

Use the URL Inspection tool on individual URLs to see the HTML Google received, the canonical it selected and when it last crawled the page.

One workflow that saves hours is an indexation map: a spreadsheet listing each URL type, whether it should be indexed, its canonical target and whether it belongs in the sitemap. It stops teams undoing each other's decisions.

3. Rendering and JavaScript

Google processes JavaScript sites in three phases: crawling, rendering and indexing. Pages are queued for rendering separately, so content that only appears after JavaScript runs may be processed later than content in the initial HTML.

Google's own documentation is clear that it has rendered JavaScript for years, and in March 2026 it removed older guidance implying that loading content with JavaScript makes things harder for Search. The practical risks are narrower and more specific:

  • Noindex in the HTML: if the initial HTML carries noindex, Google may skip rendering entirely, so JavaScript that later removes it may never run.
  • Non-200 pages: pages returning a non-200 status code may not be sent to rendering at all.
  • Canonical changes: canonicalisation happens before and after rendering, so set the same canonical in the raw HTML and the rendered page, or leave it out of the raw HTML.
  • Links: Google follows real <a href> links. Navigation built on click handlers without an href may not be discovered.
  • File size: Googlebot fetches the first 2MB of an HTML file (uncompressed) and each referenced resource is bound by the same limit. Very large inline scripts or data blobs can push content past the cut-off.

Server-side rendering or static generation, which frameworks like Next.js do by default, avoids most of these problems. If you are moving from a client-rendered or WordPress setup, our guide to a WordPress to Next.js migration without losing rankings covers the rendering decisions in detail.

Get plain-English guides like this in your inbox.

One short email a month. WordPress, Shopify, SEO, no fluff. Unsubscribe in one click.

We never share your email.

A clean structure helps Google understand topic clusters and helps people move from a broad page to a specific answer.

  • Keep important pages within three or four clicks of the homepage. Deeper pages tend to be crawled less often.
  • Use breadcrumbs from category to subcategory to product or article, and keep them consistent across templates.
  • Add contextual links inside body copy, especially from established pages to newer or deeper ones.
  • Find orphan pages, URLs with no internal links, by comparing a crawl against your sitemap and Search Console exports.
  • Make pagination crawlable with real links to each page, not just a "load more" button.

Internal links are the one ranking lever you fully control. We cover how to plan them in the power of internal linking for SEO and UX.

5. Core Web Vitals: the current thresholds

Web performance dashboard showing page speed and Core Web Vitals metrics

Core Web Vitals are Google's real-user measures of loading, responsiveness and visual stability. Interaction to Next Paint (INP) replaced First Input Delay (FID) as the responsiveness metric in March 2024. The "good" thresholds, measured at the 75th percentile of page loads, are:

  • Largest Contentful Paint (LCP), loading: 2.5 seconds or less.
  • Interaction to Next Paint (INP), responsiveness: 200 milliseconds or less.
  • Cumulative Layout Shift (CLS), visual stability: 0.1 or less.

Google is careful to say that good Core Web Vitals do not guarantee top rankings. They are part of page experience, and relevance still wins. Where they matter is in close contests and in conversion, because slow, jumpy pages lose people before they read anything.

Fixes you can ship

  • LCP: compress and resize the hero image, serve AVIF or WebP, preload the LCP image, and do not lazy-load anything above the fold.
  • INP: break up long JavaScript tasks, defer non-critical scripts, and audit third-party tags. Chat widgets, heatmaps and ad scripts are frequent offenders.
  • CLS: set width and height on images and embeds, reserve space for banners and ads, and use font-display: swap with preloaded fonts.

Use PageSpeed Insights for real-user (CrUX) data and Lighthouse for lab debugging. The Core Web Vitals report in Search Console groups URLs by template, so fix at the template level rather than page by page. For a worked example on e-commerce, see Core Web Vitals for Shopify stores.

6. Mobile

Google uses the mobile version of a page for indexing and ranking, and finished moving all sites to mobile-first indexing in 2024. Mobile bugs are SEO bugs.

  • Serve the same primary content, headings and structured data on mobile as on desktop. Content hidden from mobile templates may effectively not exist.
  • Set a correct viewport meta tag and make sure text is readable without zooming.
  • Avoid intrusive interstitials that cover the main content on arrival.
  • Make sure internal links in mobile navigation are real, crawlable links.

Search Console's old Mobile Usability report has been retired, so test with Lighthouse and by actually using the site on a phone.

7. HTTPS and security

HTTPS is a baseline expectation. Check that:

  • Every HTTP URL 301 redirects to its HTTPS equivalent in one hop.
  • There is no mixed content (HTTP images or scripts on HTTPS pages).
  • Certificates auto-renew, and you get alerted before expiry.
  • The Security issues and Manual actions reports in Search Console are empty.

Hacked pages and injected spam links are a technical SEO problem as much as a security one. Google's spam policies also expanded in 2026 to cover "back button hijacking", so audit any script that interferes with the browser back button.

8. Structured data: what still earns rich results in 2026

Structured data helps Google understand what a page is about, and for supported types it makes the page eligible for rich results. The list of supported features has been shrinking, so it is worth knowing what has gone:

  • FAQ rich results stopped appearing in Google Search from 7 May 2026, and Google removed the documentation in June 2026. They had already been limited to well-known government and health sites since 2023.
  • HowTo rich results were removed in 2023.
  • In 2025 Google phased out several lesser-used types, including Course Info, Claim Review, Estimated Salary, Learning Video, Special Announcement and Vehicle Listing, followed by Practice Problem. Dataset markup is now only used by Dataset Search.

What still earns enhanced results includes Product and merchant listings (prices, availability, shipping and returns), review snippets, Breadcrumb, Article, Organization, LocalBusiness, Event, Recipe, Video, Job Posting, Discussion Forum and Q&A pages. Google keeps adding properties to these, for example category for merchant listings and new forum properties in 2026.

Rules that do not change:

  • Markup must describe content visible on the page. Ratings that are not shown, or reviews you wrote yourself, risk a manual action.
  • Validate with the Rich Results Test, and check the enhancement reports in Search Console after every release.
  • Removing markup for a retired feature is optional. Google said the 2025 phase-out does not affect rankings, the enhanced display simply stops appearing.

For the full picture on which schema types are worth implementing, read our guide to structured data, rich results and AI citations.

9. Redirects and site migrations

Redirects are where rankings are most often lost, usually during a redesign, a platform change or a domain move.

  • Use 301 (or 308) redirects for permanent moves.
  • Redirect each old URL to its closest equivalent, not everything to the homepage.
  • Remove chains. A to B to C should become A to C, and internal links should point at the final URL directly.
  • Keep redirects in place for at least a year, and ideally permanently for URLs with backlinks.

For domain moves, Google's site move guidance was updated in June 2026 to recommend using the Change of Address tool for every host variant, including www and non-www. Before any migration, build a full URL map from your crawl, sitemap, analytics and backlink data, then crawl the old URL list after launch to confirm every one lands on a 200.

10. International SEO and hreflang

If you serve more than one country or language, hreflang tells Google which version to show to which audience. Keep it brief and correct:

  • Every hreflang annotation must be reciprocal: if the UK page points to the US page, the US page must point back.
  • Point hreflang at canonical, indexable URLs only.
  • Use valid language and region codes, for example en-gb and en-us, plus an x-default for everyone else.

This is the area with the most noise, so here is what Google's documentation actually says as of October 2026.

There are no extra technical requirements for AI Overviews or AI Mode. To appear as a supporting link, a page must be indexed and eligible to show with a snippet. The same technical SEO fundamentals apply, which is why this checklist matters more, not less, in AI search.

Controls that actually work for Google Search:

  • Robots.txt rules for Googlebot control crawling for Search, including its AI features.
  • nosnippet, data-nosnippet, max-snippet and noindex limit what can be shown from your pages, including in AI features.
  • Search Console now has a Search generative AI control (under Settings), rolled out to all sites on 31 August 2026. It lets you include or exclude your site's links and content from AI Overviews, AI Mode and generative features in Discover. Inclusion is the default, and Search Console also has a generative AI performance report to help judge the trade-off.

What Google-Extended does and does not do: it is a robots.txt token that controls whether Google can use your content to train future Gemini models and for grounding in Gemini Apps and the Vertex AI API. Google states it does not affect Google Search.

What you can ignore for Google: Google's generative AI optimisation guide says llms.txt files and other "special" AI markup are not used by Google Search, and will neither help nor hurt rankings. It also says you do not need to break content into tiny chunks for AI.

Other AI companies run their own crawlers with their own robots.txt tokens, such as OpenAI's GPTBot and OAI-SearchBot. Decide deliberately which you allow, check your CDN or firewall is not blocking bots you want, and review server logs to see who is actually crawling. If you want to measure where you show up, our guide to tracking visibility in Google AI Overviews covers the practical options.

12. Tools and a 30-minute technical SEO audit

Technical SEO action plan infographic showing audit priorities

You do not need an enterprise platform. Google Search Console and PageSpeed Insights cover indexing, crawling and Core Web Vitals. A desktop crawler such as Screaming Frog (free up to 500 URLs) or Sitebulb finds status codes, redirects, canonicals and orphan pages. Add the Rich Results Test for markup, and server logs for larger sites.

The 30-minute routine

Run this monthly, and after every significant release:

  1. Five minutes, Search Console Pages report. Has the count of indexed pages moved? Any new "not indexed" reasons? Spot-check two key URLs in URL Inspection.
  2. Five minutes, sitemaps. Submitted versus indexed per sitemap. A widening gap points at the problem section.
  3. Ten minutes, crawl. Crawl the site and filter for non-200 status codes, redirect chains, canonical mismatches, noindexed pages in the sitemap and pages deeper than four clicks.
  4. Five minutes, Core Web Vitals. Check the Search Console report for templates in "Poor" or "Needs improvement", and run PageSpeed Insights on your top landing page.
  5. Five minutes, hygiene. Robots.txt unchanged, HTTPS redirects working, no manual actions or security issues, enhancement reports clean.

Then validate the whole path end to end on ten important URLs. Can Google crawl them, render the key content, index the canonical, and rank the page you actually want shown? If any step breaks, fix that before you publish more content.

Prioritise by impact, not interest: first anything blocking indexing of money pages, then duplication and crawl waste, then template-level performance and internal links, and structured data last. If you would rather have someone else run the audit and prioritise the fixes, that is what our SEO services team does every week.

The short version

Technical SEO is about access and clarity: can Google crawl, render, index and understand the pages you want to rank? Start with the index, fix what blocks your money pages, then work through performance, structure and markup. Repeat the 30-minute routine monthly so fixes do not quietly regress.