Processing Background Jobs in Drupal with the Queue API
When a Drupal site starts handling thousands of users across time zones, certain operations simply cannot run inline anymore. Mailing a newsletter, resizing uploaded photos, syncing records with an external CRM, or generating PDF invoices all benefit from being deferred to a separate process. The Queue API in Drupal Core gives site builders a standardised way to push tasks onto a backlog and let dedicated workers chew through them without making visitors wait. For teams in Australia running sites that serve both Sydney customers at nine in the morning and Perth clients finishing their lunch, this asynchronous approach also smooths out traffic patterns and keeps the main request cycle lean.
Despite its reputation as a "Core" feature, the Queue API is one of the more underused tools in the typical Drupal developer's kit. Many engineers reach for contributed modules like BackgroundProcess or hook_cron directly, missing how the queue abstraction offers retries, pluggable storage, and per-worker logic out of the box. This article walks through how queues are structured, how to write a custom worker, and how to operate them reliably in production environments hosted around Melbourne, Brisbane, and beyond.
How Drupal's Queue API Works
At its heart, a queue in Drupal is a FIFO list of items that have been enqueued and are waiting to be consumed. Items are usually associative arrays or small objects, and they live in a backend that can be swapped without changing application code. The default backend in a stock Drupal install is the database, but the API also supports pluggable alternatives such as the Redis-backed module or any custom service implementing QueueInterface.
When you call \Drupal::queue('my_queue')->createItem($payload), Drupal serialises the payload and stores it in the appropriate table or remote service. A QueueWorker plugin, identified by the same machine name, is then responsible for processing each item. The plugin's processItem() method receives the deserialised payload and contains the actual business logic. Because the worker is a normal plugin, you can attach dependencies through create() and benefit from the same service container that powers the rest of the site.
The separation between the queue backend and the worker plugin is what makes the system flexible. A team in Adelaide might run a database queue during development, switch to Redis for staging, and keep the same worker code untouched. As long as the contract between createItem() and processItem() stays consistent, the underlying store is a deployment concern rather than an architectural one.
Comparing Common Queue Backends
| Backend | Persistence | Best fit | Notes |
|---|---|---|---|
| DatabaseQueue | MySQL/PostgreSQL table | Small sites, dev environments | Bundled in Core, no extra infrastructure |
| ReliableQueue (database, claimed) | Same DB but uses BEGIN/COMMIT | Sites needing safer claim semantics | Avoids duplicate processing across restarts |
| RedisQueue (contrib) | Redis server | High-throughput production sites | Lower latency, supports blocking pops |
| Amazon SQS / Beanstalkd | Cloud or self-hosted service | Multi-server fleets | Decouples workers from web tier entirely |
For most Australian agencies hosting with local providers in Sydney or Melbourne, the database backend is the path of least resistance during the build phase. The moment the site moves to a load-balanced setup with more than two web nodes, the conversation usually shifts to Redis. SQS becomes attractive when workers need to run on completely separate infrastructure, such as a dedicated batch node in a different availability zone.
A practical rule of thumb is to start with the database, then graduate to a dedicated broker only when monitoring shows the queue table is competing with regular content queries. Premature optimisation, in this case, often adds operational complexity without measurable gain for sites that process fewer than a few thousand jobs an hour.
Writing a Custom Queue Worker
To create a worker, you define a plugin under /modules/custom/my_module/src/Plugin/QueueWorker/. The annotation includes the id, label, and an optional cron time limit that tells the runner how many seconds it may spend in a single pass. Inside processItem() you receive the deserialised payload and you can call any service you need, from the entity type manager to a custom HTTP client.
A practical example is processing CSV imports uploaded through a form. Instead of blocking the user for thirty seconds, the form submit handler validates the file, calls createItem() with a row of metadata, and returns a confirmation message. The worker then streams the CSV in chunks, imports each row, and updates a state value so the progress bar on a status page can read the current position. Failures are caught with a try/catch block, and uncaught exceptions are written to the watchdog log, which Drupal retains for site admins to inspect.
Because the worker is a plugin, you can version it cleanly. If the payload format changes, a new worker with a new id can consume a different queue, leaving the old queue to drain naturally. This pattern keeps upgrades safe even when you cannot afford downtime, which is a real concern for retailers running Boxing Day sales across the country.
Cron, Timing, and Time Zone Realities
Drupal's cron typically runs every three hours, but it can be triggered more aggressively on managed platforms. The Queue API hooks into this through CronQueueSuspendDelay and the cron worker that pops items off every queue whose plugin opted in via the cron annotation. Items that exceed the cron time limit remain in the queue for the next pass, which keeps request workers from timing out.
A subtle but important point for Australian teams is that Drupal stores time in UTC internally and respects the site's default_timezone configuration. A site configured for Australia/Sydney will see cron "last run" timestamps in AEDT or AEST depending on daylight saving. When troubleshooting stuck queues, it is worth checking that the cron runner's clock matches what the database thinks, because mismatches often cause queues to appear empty when they are simply being checked too eagerly.
For more granular control, the Ultimate Cron module replaces Drupal's single cron hook with per-queue scheduling, allowing a worker to fire every minute while a heavy export job runs hourly. Many Drupal shops in Brisbane use this approach to keep integration jobs separated from interactive ones. If a particular worker is regularly hitting its time limit, bumping the cron budget via annotation or splitting the work into smaller queue items is usually a better fix than raising PHP's max_execution_time, which only masks the underlying problem.
Reliability, Retries, and Failure Modes
A queue is only as useful as its failure handling. Drupal's default behaviour is to delete an item as soon as a worker successfully claims it. If the worker crashes mid-process, the item is lost forever in the database backend, which is why the "claimed" sub-queue concept was introduced for the ReliableQueue backend. With ReliableQueue, claiming an item and processing it happen inside a single transaction, so a worker that dies between the two steps will have the item returned to the queue at the end of the request.
For more granular retry logic, many developers wrap their worker logic in an explicit loop with a counter, throwing a custom exception after a certain number of attempts. Items that have failed too many times can be moved to a "dead letter" queue for manual review. This is especially relevant for compliance with the Australian Privacy Principles, where data exports and deletion jobs must be auditable, not just best-effort.
Logging deserves attention here as well. Watchdog, or its modern replacement in the syslog module, captures every exception by default, but the volume of noise can bury real problems. Adding structured log entries with the queue name and a hash of the payload makes it possible to grep for a specific failing item, which is invaluable when a site receives a complaint from a customer in Perth and you need to trace a specific job. For teams operating under the Essential Eight maturity model, structured logging also supports the detection and response controls the ACSC recommends, and queue endpoints benefit from the same upstream hardening as the rest of the application tier - the Radware Cloud Armor setup walkthrough shows how that layer typically sits in front of a busy job pipeline.
Scaling Workers and Operational Habits
When a site outgrows a single cron pass, the natural next step is to run dedicated workers. A simple supervisor setup with systemd or a small Kubernetes job can pop items off faster than cron ever could. The catch is that workers must be idempotent, meaning the same item processed twice should produce the same end state. Email sends, payment captures, and order status updates all need this property, otherwise a flaky network can double-charge a customer on the other side of the country.
Monitoring is the other half of running queues well. Drupal exposes queue sizes through the State API, and a tiny custom service that ships those counts to Prometheus or Datadog turns a black box into a dashboard. A spike in queue depth on a Saturday afternoon in Melbourne, when local traffic is low, is a strong signal that a worker has stalled. Conversely, a queue that drains faster than it fills usually means the trigger logic is misconfigured and is pushing too few items onto the stack.
For teams supporting multiple Drupal properties under one roof, it pays to standardise on a single queue infrastructure pattern, document the worker plugin ids in a runbook, and review the deferred jobs during post-incident reviews. Reach out through the Drupalwoo contact page if you would like a starter template for a claim-and-retry worker, or if you need help integrating deferred jobs into an existing infrastructure.