Guides · 4 min read · Updated 2026-09-08

Duplicate content: how to find it, why it happens, and how to fix it without losing rankings

Exact and near-duplicate pages split rankings and waste crawl budget. Where duplicates come from on real sites, how to detect them properly (main content only), and the fix for each cause.

Duplicate content isn’t a penalty; it’s a dilution. When the same content lives at several URLs, search engines pick one to rank and ignore the rest, links to the others are partly wasted, and crawl budget is spent fetching the same words repeatedly. On most sites the duplicates are accidental, created by the platform rather than by anyone copying anything, and the fix is a canonical tag or a redirect rather than a rewrite.

Where duplicates come from

URL variants of the same page. http and https, www and bare, trailing slash and none, uppercase and lowercase paths, index.html, tracking parameters like utm_source, session identifiers. Each is a distinct URL to a crawler. A single product can exist at sixteen addresses without anyone noticing.

Faceted navigation and sorting. ?colour=blue&sort=price produces a page that is mostly the base category with a different order. Multiply by every combination of filters and a catalogue of 500 products has 50,000 crawlable URLs.

Pagination. Page 2 of a category shares the header, the intro paragraph and the sidebar with page 1, and differs only in which products are listed.

Templates that outweigh content. A location page with two sentences of unique text and 900 words of shared boilerplate is a near duplicate of every other location page.

Syndication and republishing. A press release on your site and on the wire; a blog post republished on Medium; product descriptions copied from the manufacturer that every retailer also uses.

Print, AMP and mobile versions. Separate URLs for the same content.

Staging and development sites that are publicly reachable and indexed.

Detecting it properly

The naive method, hashing the whole HTML, is wrong in both directions. It misses pages that differ by a timestamp or a session token, and it flags documentation sites where a huge shared sidebar makes every page look like every other page.

Compare the main content only. Strip navigation, header, footer and sidebars, then compare what’s left. Truelint extracts the main content using the page’s own landmarks (<main>, <article>, role="main") and falls back to removing the chrome, so the sidebar never counts.

Detect near duplicates, not just exact ones. Two pages that share 90% of their sentences are duplicates for ranking purposes. Truelint fingerprints the main text with a similarity hash and reports pairs above a threshold, in bounded memory even on large sites.

Look at titles and H1s separately. Duplicate titles across different pages are the most common finding on every audit and are cheap to fix; they often reveal duplicate pages you didn’t know about.

Cross-check with the canonical tab. If duplicates already canonicalise to one of them, they’re handled. The finding you want is duplicates with no canonical, or canonicals that point at each other.

Fixing it, by cause

URL variants: pick one canonical form and 301 every other form to it at the server level. Then set self-referencing canonical tags so any variant that slips through still resolves. Configure the CMS to generate links only in the canonical form.

Tracking parameters: canonicalise to the parameterless URL. Don’t block them in robots.txt; a blocked URL can still be indexed from external links and now can’t read its own canonical.

Facets and sorting: decide which combinations have search demand (usually one filter deep: “blue shoes”, not “blue size-9 leather shoes under £100”). Make those real pages. Canonicalise everything else to the base category, or noindex it. Keep them crawlable so the products get found.

Pagination: give each page a self-referencing canonical, not a canonical to page 1, which hides the products on later pages. Use rel="next" and rel="prev" for clarity even though Google no longer uses them as a signal. Consider a “view all” page if the list is short enough.

Thin templates: write the unique part. If a location page can’t say anything specific about that location, it shouldn’t exist as a separate URL; fold it into a regional page.

Syndication: the republished copy should carry a canonical back to your original. Where you can’t control that, publish on your site first and let the copies follow.

Manufacturer descriptions: rewrite for your best-selling products; the long tail can keep the shared text with a canonical to your own page.

Staging: password-protect it. A noindex is not enough; the site should not be reachable.

What not to do

• Don’t rely on robots.txt to hide duplicates. Blocked pages can still rank on the strength of links, and they can’t pass a canonical.
• Don’t canonicalise to a page that’s noindexed or that itself canonicalises elsewhere. Chains and contradictions get ignored.
• Don’t 301 paginated pages to page 1. You’ll lose the deep pages’ products from the index.
• Don’t rewrite content just to be different. Near-duplicate location pages with a synonym swapped are still duplicates to a modern search engine, and worse for users.

Checking your own site

In Truelint, the Content tab has filters for exact duplicates, near duplicates and thin pages, computed on the main content. The Canonicals tab shows which of them already resolve. Fix the causes by template, re-crawl, and compare the two crawls; the duplicate count should fall to the number of genuine variants you chose to keep.