Drupalwoo ☉

Dynamic Sitemaps with a Custom Drupal Module

Search engines use XML sitemaps as a map of URLs, but the file is only a discovery aid. It does not guarantee indexing, rankings or visibility. The useful question for a Drupal site is whether its published URLs change often enough that a hand-uploaded XML document becomes unreliable.

Creating a Drupal module that generates sitemaps dynamically keeps that map tied to published content, aliases and language settings. The implementation can remain small: a public route, a controller that queries eligible entities, XML escaping, sensible caching and an invalidation strategy. This guide uses Drupal 10 and 11 conventions and suits councils, retailers and membership sites operating under an Australian .com.au domain.

Choose between static and dynamic output

A dynamic route is not automatically superior. The best approach depends on URL volume, publishing frequency and the availability of suitable Drupal modules.

Approach Freshness Server effort Best suited to
Static XML file Updated manually or by cron Low Small sites with stable URLs
Dynamic Drupal route Reflects current content when cached Moderate Frequently updated sites
Contributed sitemap module Configurable and generated from entities Moderate Sites needing an administration interface

A dynamic route is most useful for news portals, catalogues, event listings and member directories where URLs appear, change or disappear without notice. A static file remains simpler for a small microsite. Before choosing custom code, check whether a maintained contributed XML sitemap module already covers the required URL rules; an established module usually reduces long-term maintenance compared with a bespoke implementation.

Estimate the URL count and update frequency as well. The sitemap protocol limits one file to 50,000 URLs and 50 MB uncompressed. Large Drupal estates should publish a sitemap index and smaller child files rather than forcing one controller response to carry every entity.

Define a predictable public route

Create the module under web/modules/custom/dynamic_sitemap/ and give its controller the \Drupal\dynamic_sitemap namespace. A stable path such as /sitemap.xml is easy for search engines, crawlers and administrators to remember. It should accept no query parameters and have one predictable response.

# dynamic_sitemap.routing.yml
dynamic_sitemap.sitemap:
  path: '/sitemap.xml'
  defaults:
    _controller: '\Drupal\dynamic_sitemap\Controller\SitemapController::build'
  requirements:
    _permission: 'access dynamic sitemap'

Define access dynamic sitemap in dynamic_sitemap.permissions.yml and grant it to the anonymous role. Without that permission, search engine crawlers will receive a 403 response. The controller can inject Drupal's entity type manager, URL generator and logger, then return a Symfony Response with an application/xml content type.

Each listed URL must be a canonical, indexable destination that returns a successful response. Drupal aliases sometimes redirect to canonical entity routes, so list the final destination rather than the redirecting alias. Remove deleted pages promptly, as a sitemap containing persistent 404 or 410 responses wastes crawler activity and obscures genuine indexing problems.

Do not treat the sitemap as private infrastructure. Staging environments should return a 404 response or use suitable noindex headers, and any authenticated area must remain protected by Drupal access controls. A sitemap must never contain draft records, internal search results or URLs reserved for administrators.

Collect entities without exposing private content

A node query can provide the eligible records, but the module should make its selection rules explicit. This example limits the result to published, accessible nodes and the first protocol-sized batch.

$ids = $this->entityTypeManager->getStorage('node')->getQuery()
  ->accessCheck(TRUE)
  ->condition('status', 1)
  ->sort('changed', 'DESC')
  ->range(0, 50000)
  ->execute();

$nodes = $this->entityTypeManager->getStorage('node')->loadMultiple($ids);

Use access-aware queries and verify that every configured content type has a stable canonical route. Load entities in batches to avoid exhausting PHP memory on a large catalogue. Filter out entities marked noindex, URLs containing fragments and internal paths that search engines cannot reach directly.

Multilingual sites need a deliberate policy. One sitemap may contain canonical URLs for every language, while separate language domains or installations usually need separate files. Translated URL pairs should only receive hreflang annotations when the alternates are reciprocal and correctly implemented; simply listing both URLs provides weaker guidance.

Australian businesses frequently maintain location pages for Sydney suburbs, Melbourne branches and Brisbane depots. Generate one canonical URL for each physical location rather than separate pages for minor address variations. If a Drupal page draws on public transport data, list the canonical Drupal landing page rather than both it and the external source unless each page is independently owned and indexable.

Escape every URL before placing it inside the XML document. Ampersands and other reserved characters must remain valid XML, and an unescaped value can invalidate the entire response.

$escapedUrl = htmlspecialchars(
  $url,
  ENT_XML1 | ENT_QUOTES,
  'UTF-8'
);

Use the entity's changed timestamp only when it represents a meaningful content update. Routine workflow changes can make lastmod less useful. Drupal stores timestamps consistently, but converting them to the site's Australia/Sydney timezone still requires care during daylight-saving transitions. Store the value as UTC and format it according to the W3C date specification.

Cache responses and connect crawler discovery

Rebuilding the complete XML document for every crawler request can place unnecessary pressure on Drupal, especially when thousands of pages are published. Cache the response in Drupal's cache layer or at the CDN, using a shared key for the site's domain and language configuration.

A suitable starting header is:

Cache-Control: public, max-age=900, s-maxage=3600, stale-while-revalidate=86400

The short browser lifetime gives administrators a reasonable update window, while the shared-cache lifetime reduces repeated generation. The sitemap must remain identical for every visitor, so it should never include authenticated or personalised URLs.

Invalidate cached output when content is published, unpublished or deleted, or when an alias, translation or canonical route changes. Drupal entity hooks or a dedicated cache-tag service can trigger this work. A scheduled rebuild is a useful fallback, but relying only on cron leaves search engines with stale information between runs and creates unnecessary timing dependencies.

Advertise the sitemap from robots.txt, using the site's complete public address:

User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com.au/sitemap.xml

Adjust the path if Drupal is installed in a subdirectory. Robots directives control crawling rather than access, so they are not a substitute for permissions. Search engines can still discover a blocked URL through external links, which is why private records need stronger protection and should be excluded from the sitemap itself.

Production checks for Australian Drupal sites

Test the module with unit tests for entity filtering, URL escaping, language handling and canonicalisation. Integration tests should request the route as an anonymous visitor and verify the status code, content type and XML structure. After deployment, submit the sitemap to Google Search Console and Bing Webmaster Tools, then monitor crawl errors, generation time and the number of discovered URLs.

Australian users commonly search on phones while commuting or looking for nearby services, particularly across Sydney, Melbourne and Brisbane. A current sitemap supports discovery, but it does not replace responsive rendering, accurate local pages or fast hosting. For organisations covered by Australian consumer law, price and availability claims must also remain truthful. Many Australian businesses are subject to the Privacy Act 1988 and the Australian Privacy Principles, so public accessibility does not automatically justify including personal profiles, order histories or draft member records.

These production safeguards keep the generated sitemap useful and appropriately scoped:

Treat the sitemap as a public contract between Drupal and external platforms. A documented route, tested selection rules and deliberate invalidation process allow the module to remain accurate while the site's content, structure and Australian market presence continue to change.