Crawling: make important URLs easy to discover
Search engines discover pages through links, sitemaps and previously known URLs. Important pages should be reachable through normal HTML anchors, not only through JavaScript interactions, internal search forms or orphaned sitemap entries. Keep the architecture shallow enough that a visitor can navigate from a hub to detailed pages without guessing.
Robots.txt should control crawling, not be used as a substitute for noindex. Blocking a URL can stop a crawler from seeing a page-level noindex directive. Reserve robots rules for areas that do not need to be fetched at all, such as private storage or internal endpoints.
Indexing: send one clear canonical signal
Duplicate URLs fragment signals and waste crawl attention. Use redirects for permanent moves and canonical tags when duplicate or near-duplicate versions must remain accessible. Canonicals should be absolute, self-consistent and point to an indexable 200 URL.
Keep XML sitemaps clean. Include only canonical pages you want indexed, provide accurate last-modified dates, and remove URLs that are drafts, private tools, search results or redirects. A sitemap helps discovery; it does not force indexing.
Rendering, JavaScript and content visibility
Critical content, links and metadata should be present in the server-rendered HTML whenever practical. Google can render JavaScript, but rendering adds another processing step and can fail when scripts or resources are blocked. For service pages and articles, there is usually no SEO advantage in hiding core text behind client-side rendering.
Test representative URLs with Search Console’s URL Inspection tool. Verify the fetched HTML, rendered content and canonical Google selected. The live test is especially useful after routing, CDN, security or framework changes.
Status codes, redirects and error handling
Return 404 or 410 for genuinely missing pages, 301 for permanent moves and 200 only when the requested page exists. Soft 404s—pages that look missing but return 200—confuse indexing. Avoid redirect chains and loops because each hop adds latency and makes migrations harder to interpret.
During redesigns, build a URL map before launch. Redirect each important old URL to its closest relevant replacement instead of sending everything to the homepage.
Structured data and page experience
Structured data helps search systems understand entities and page types, but it must match visible content. Use organization markup for the business, Article or BlogPosting for editorial content, BreadcrumbList for hierarchy and Service where a service page clearly describes an offering.
Performance is also part of technical quality. Aim for stable layouts, fast largest-content rendering and responsive interactions. Google recommends good Core Web Vitals as part of an overall strong page experience. Measure real-user data when enough traffic exists, then use lab tools to diagnose individual bottlenecks.
Reference: Google Core Web Vitals guidance.
Common questions.
Does submitting a sitemap guarantee indexing?
No. A sitemap helps search engines discover canonical URLs, but indexing still depends on accessibility, duplication, content value and other signals.
Should robots.txt block pages that use noindex?
Usually no. If a crawler is blocked from fetching the page, it may not see the noindex directive.
What is the first technical SEO check after a deployment?
Confirm important URLs return 200, canonical tags point correctly, robots directives are indexable, the sitemap fetches successfully and internal links resolve without redirects or errors.
