In this article
The most useful XML sitemap best practices come down to three habits. List only canonical, indexable URLs that return a 200 status. Split large sites into segmented sitemaps. Keep lastmod dates honest so search engines trust your signals. Get those right and Googlebot spends its crawl time on the pages that earn you traffic, not on redirects, duplicates or thin filter pages. The guide below covers structure, segmentation, submission and ongoing maintenance, with the specific limits and mistakes I see most often in technical audits.
What Should an XML Sitemap Actually Include?
An XML sitemap should include only URLs you want indexed. Each one should be canonical, return a 200 status code, carry no noindex tag and not be blocked by robots.txt. Leave out redirects, 404s, parameter variants, paginated duplicates and staging links. The file is a list of recommendations, so every weak entry makes the rest of the list less trustworthy.
Many teams treat the sitemap as an inventory of everything the CMS can produce. That is a mistake. Google has said plainly that a sitemap is a hint, not a directive. Hints work better when they are consistent with your other signals.
Here is a quick filter I run on every URL before it goes in:
- Status code: it must be 200. No 301s, 302s, 404s or soft 404s.
- Canonical tag: it must be self-referencing. If page A canonicalizes to page B, list only B.
- Indexability: no meta robots noindex and no X-Robots-Tag noindex header.
- Crawlability: not disallowed in robots.txt.
- Exact URL match: same protocol, same host, same trailing-slash convention as the canonical.
That last point catches people constantly. A site canonicalizes to /shoes/ with a trailing slash, while the sitemap generator outputs /shoes without one. Every entry then points to a redirect. Search Console won't always flag it loudly, but you've quietly turned your sitemap into a list of hops.
Does every indexable page need to be listed? Ideally, yes. Your sitemap should mirror the set of pages you'd be happy to see in search results, no more and no less. If you're still defining that set, the on-page SEO checklist is a good place to decide which pages deserve to be indexable at all.
Respect the Technical Limits and Formatting Rules
The sitemaps.org protocol, which Google, Bing and other major engines follow, sets hard limits. A single sitemap file may hold up to 50,000 URLs or 50 MB uncompressed, whichever you hit first. A sitemap index file can reference up to 50,000 child sitemaps. Go past either limit and the file may be partially read or rejected.
One gotcha: compression doesn't buy you extra room. You can gzip a sitemap to save bandwidth, and engines accept .xml.gz files happily. The 50 MB cap still applies to the uncompressed size. I've seen a 9 MB gzipped file that unpacked to well over the limit, and nobody noticed for months.
Formatting rules matter too:
- Encode the file as UTF-8.
- Use absolute URLs, including the protocol (https://).
- Escape special characters. An ampersand in a URL must be written as & or the XML breaks.
- Declare the correct namespace in the urlset element, plus any extension namespaces for images, video, news or hreflang.
- Keep each sitemap on the same host it describes, unless you've verified both properties in Search Console for cross-site submission.
Honestly, I wouldn't push files anywhere near 50,000 URLs. Smaller files of 10,000 to 25,000 URLs are easier to debug, regenerate faster and give you more granular reporting. The limit is a ceiling, not a goal.
Validate before you ship. Paste a sample into an XML validator, or run the file through Screaming Frog's sitemap crawl mode. A single unescaped character can invalidate an entire file. Then Search Console reports a parsing error, and thousands of URLs lose their discovery path overnight.
Segment Sitemaps by Template and Business Value
Segmentation is the practice that separates a functional sitemap from a genuinely useful one. Instead of one giant file, split URLs into logical groups and tie them together with a sitemap index. Search engines don't rank you higher for this. The payoff is diagnostic.
Why? Search Console reports indexing coverage per submitted sitemap. If you segment by page type, you can see at a glance that 96% of your category pages are indexed while only 41% of product pages are. That gap tells you exactly where to look. One monolithic file hides it.
A segmentation scheme I've used on e-commerce sites with around 200,000 URLs:
- sitemap-categories.xml for top-level and subcategory pages
- sitemap-products-1.xml, -2.xml and so on for product detail pages, chunked
- sitemap-blog.xml for editorial content
- sitemap-static.xml for about, contact and policy pages
- sitemap-images.xml when product imagery drives search traffic
Publishers can segment by date instead, say one file per month of articles. That keeps older files stable and the newest file small. Stable files matter. If a file hasn't changed, crawlers have little reason to keep refetching it.
Another trick: create a temporary "priority" sitemap for a batch of new or recently updated pages you want discovered quickly, such as a product launch or a migrated section. Submit it separately and watch its indexing ratio climb. Once those URLs are indexed, fold them back into the regular segments.
Segmentation also pairs well with your broader content strategy. When you map sitemap groups to the sections that drive revenue, sitemap health becomes a proxy for the health of the pages your core SEO best practices are built around.
Make Lastmod Trustworthy and Skip Priority and Changefreq
Of all the tags in a sitemap, lastmod is the one that still earns its keep. Google has confirmed it uses lastmod when the value is consistently and verifiably accurate. The two other optional tags, priority and changefreq, are ignored by Google. You can drop them and lose nothing.
The word "consistently" carries a lot of weight. Google's guidance treats lastmod as the date of the last significant change to the page's main content, structured data or links. A copyright year update in the footer doesn't count. Neither does a sidebar widget refresh.
Here's what nobody tells you during a replatform: many CMS and static-site setups stamp every URL with the build timestamp on each deploy. Deploy twice a day and your sitemap claims 80,000 pages changed this morning. Crawlers figure out quickly that the dates are noise, and then they discount them entirely. You've burned the one signal that could have sped up recrawling of your real updates.
How to keep lastmod honest:
- Pull the date from a content-modified field in the database, not the file system or the build process.
- Trigger an update only on meaningful edits: body copy, price, availability, headings, key internal links.
- Use W3C Datetime format, for example 2026-11-05 or a full timestamp with a timezone offset.
- Never set future dates. It happens more than you'd think with timezone bugs.
- Match the visible "updated" date on the page where one exists.
Should you remove priority and changefreq from existing files? I would. They add bytes and confusion, and junior team members sometimes spend hours tuning priority values that Google never reads. Bing has historically said it may use them as weak hints, but lastmod and IndexNow are far stronger levers there.
How Do You Get Search Engines to Find Your Sitemap?
Reference every sitemap or sitemap index in robots.txt with a Sitemap: directive using the full absolute URL, then submit the index in Google Search Console and Bing Webmaster Tools. That covers discovery by any compliant crawler plus direct reporting. Google retired its sitemap ping endpoint in 2023, so pinging no longer helps Google find updates.
The robots.txt line is simple: Sitemap: followed by the full URL of your index file. It can sit anywhere in the file and doesn't need to belong to a user-agent group. You can list multiple sitemaps, one per line.
Submitting in Search Console gives you something robots.txt can't: reporting. You'll see when Google last read the file, how many URLs it discovered and whether there were errors. Submit the index, not each child file individually, unless you specifically want per-file stats at the top level of the report.
For Bing, Yandex and other engines that support it, IndexNow is worth setting up. It lets you push URL changes the moment they happen rather than waiting for a recrawl. Plugins for WordPress and integrations from major CDNs such as Cloudflare make it a ten-minute job. Google does not participate in IndexNow, so treat it as a complement to sitemaps, not a replacement.
One practical caution. If your sitemap sits behind a CDN, check the cache headers. I once spent an afternoon chasing "missing" new pages, only to find the CDN was serving a sitemap cached three days earlier. Set a short TTL on sitemap files, or purge them on regeneration.
Most of the platforms reviewed in our guide to choosing the right SEO tools can monitor robots.txt and sitemap availability automatically, which saves you from discovering a 500 error weeks too late.
Use Specialized Sitemaps Where They Add Value
Beyond standard URL sitemaps, the protocol supports extensions for images, video, news and alternate language versions. Use them where they solve a real discovery problem. Don't add them just because a plugin offers a checkbox.
Image sitemaps help when images load via JavaScript or sit on a CDN subdomain that crawlers might not connect to the page. Google trimmed the supported tags in 2022, deprecating caption, title, geo_location and license. Today the image:loc element is what matters. If your images are plain img tags in server-rendered HTML, you may not need an image sitemap at all.
Video sitemaps are worth the effort for sites where video is the main content of a page. They let you supply the thumbnail, title, description and content or player location, which helps Google surface the video in its results.
News sitemaps are strict. Google's documentation says to include only articles published in the last two days, with a maximum of 1,000 URLs per news sitemap. Old articles should roll off automatically. Keep them in your regular sitemap for long-term indexing.
Hreflang in sitemaps is often the cleanest way to manage international sites. Each URL entry lists all of its language alternates using xhtml:link elements, including a reference to itself. The big advantage over in-page hreflang tags is maintenance: one generated file instead of thousands of page templates. The catch is reciprocity. Every alternate must point back, or Google may ignore the pairing. Generate these entries programmatically from a single source of truth, never by hand.
Pick only the extensions your site genuinely needs. A small B2B site with 300 pages and a handful of images gains nothing from four sitemap types. A global marketplace with localized catalogs gains a lot.
Maintain, Monitor and Automate
A sitemap is not a set-and-forget file. Sites change daily. Products sell out, articles get consolidated, sections get redirected. Without maintenance, sitemaps drift until they're full of dead entries.
Generate sitemaps dynamically from your database or CMS rather than exporting static files by hand. Tie removal to the same events that change a page's status: when a product is permanently discontinued and redirected, it should vanish from the sitemap in the same release.
A monthly monitoring routine I recommend:
- Check Search Console's Sitemaps report for fetch errors and the last-read date on each file.
- Review the Page indexing report filtered by sitemap. Look at the "Crawled – currently not indexed" and "Discovered – currently not indexed" buckets per segment.
- Crawl the sitemap with Screaming Frog or Sitebulb and flag any non-200, non-canonical or noindexed URLs.
- Compare sitemap URLs against a full site crawl to find orphan pages that appear in the sitemap but have no internal links.
- Spot-check lastmod values against real edit history on a sample of 20 pages.
That orphan check deserves emphasis. A sitemap can get a page discovered, but it rarely gets a page ranked on its own. If an important URL only exists in the sitemap, with no internal links pointing to it, Google sees a page your own site doesn't vouch for. Fix the linking first.
Track the indexing ratio per segment over time: indexed URLs divided by submitted URLs. A sudden drop in one segment after a template change is an early warning that something broke. Catching it in a week instead of a quarter can protect real revenue, and it fits naturally alongside other ways to improve search rankings that depend on pages actually being in the index.
Finally, document the logic. Write down which URLs qualify, how lastmod is set and who owns the generator. When the developer who built it leaves, that page of notes is worth more than the code.
Putting These Practices to Work
Start with the cleanup. Strip every non-canonical, redirected or noindexed URL from your current files. Next, segment by template so Search Console can tell you where indexing breaks down. Then fix lastmod so it reflects real content changes. Reference everything in robots.txt, submit the index and add IndexNow for engines that support it. Revisit the files monthly. None of these XML sitemap best practices is glamorous, but together they give crawlers a clean, credible map of your best pages. That is what speeds up discovery and indexing.
Frequently Asked Questions
How many URLs should I put in each XML sitemap file?
The protocol allows up to 50,000 URLs or 50 MB uncompressed per file. In practice, files of 10,000 to 25,000 URLs are easier to manage. They regenerate faster, fail less dramatically when something breaks, and give you more granular indexing data in Search Console when grouped by page type inside a sitemap index.
Does Google still use priority and changefreq in sitemaps?
No. Google has stated it ignores both the priority and changefreq tags. It does use lastmod, but only when the dates are consistently accurate across your site. Removing priority and changefreq won't hurt your Google visibility and makes files simpler. Focus that effort on making lastmod reflect genuine content updates instead.
Should noindexed or redirected pages ever appear in a sitemap?
Generally, no. Listing noindexed, redirected or non-canonical URLs sends conflicting signals and wastes crawl attention. The one short-term exception is during a migration, when some SEOs keep old URLs listed briefly so crawlers find redirects faster. Remove them once the new URLs are indexed, usually within a few weeks.
Can a sitemap fix pages stuck in Discovered – currently not indexed?
Rarely on its own. That status means Google knows the URL exists but hasn't prioritized crawling it, often due to weak internal linking, thin content or crawl budget limits on large sites. Strengthen internal links, improve page quality and confirm accurate lastmod values. The sitemap helps discovery; linking and quality drive the indexing decision.
