Processing large data sets in Drupal with the Batch API

A Melbourne agency running a Drupal commerce site for an outdoor retailer ran into a familiar wall last summer. They had booked a migration of more than 80,000 product nodes, but PHP ran out of memory halfway through every attempt. After they rebuilt the workflow around the Batch API the same job finished cleanly across three afternoons, with progress visible to the client. That pattern comes up again and again across the Australian Drupal community, whether the workload is a one-off import or a long-running cleanup.

Large data sets in Drupal usually mean anything from a few thousand nodes upwards, and they show up everywhere: a content audit for a Brisbane council, a media library migration for a university in Perth, or a bulk taxonomy reclassification after a rebrand. PHP's default execution ceiling will cap a single request at around thirty seconds even on a generously tuned hosting plan, and shared environments trim that further. The Batch API is the framework that ships inside Drupal core for getting around that limit.

What the API actually does is split a heavy job into a chain of small operations, each finishing well under the PHP timeout. After every slice Drupal queues the next slice and reloads the page, so a browser session stays responsive and a Drush command keeps churning through data on a staging server. The framework tracks state, re-runs any failed slice, and offers hooks so you can surface progress to the user.

What follows is a practical walkthrough drawn from real batch jobs: starting with the structure of a batch operation, then module code, progress reporting, Drush automation, and finally the performance tweaks that make the difference between a slow batch and a fast one.

The shape of a batch operation

The Batch API is built around three collaborating pieces: an array of operation definitions, a finished callback that runs when the chain ends, and an optional progress callback that updates the UI mid-run. Each operation in the array is itself a small array with a callback function, an arguments list, and a label Drupal uses in its progress messages.

You can think of each operation as a single transaction. Drupal calls it, hands over the arguments you declared, waits for it to return, and schedules the next operation by triggering an internal batch endpoint over HTTP. Because the process is request-driven, it survives across page loads and even sessions if you store the batch id, which is handy when a long cleanup is left overnight on a staging server in Sydney or Adelaide.

The finished callback decides what happens once every operation has completed or after a fatal error. It receives the batch context array, which carries any state you have piped between operations, plus an indicator of whether the run finished cleanly. Most Australian teams keep this callback simple: a watchdog log line, a redirect to a results page, and an email notification to the project lead back in Melbourne.

A progress callback is optional but worth wiring up for any job a user watches in real time. It runs after each operation and exposes the total count, the count processed so far, and a label you can render onto the page. Without it the UI just shows a spinner, which feels ancient on a modern Drupal admin theme.

Building a batch job in a custom module

In a custom module the practical starting point is a submit handler on a form that sets up the operations array and returns it. Drupal picks up the returned array and hands control to the batch processor, so the rest of the page load becomes a redirect to the batch endpoint. We will call our module migrate_cleanup and expose the form at /admin/content/migrate-cleanup.

Listing the operations is usually a loop over an entity query. Since each operation should only handle a small slice, a typical pattern is to chunk the results into groups of fifty or one hundred. That keeps memory predictable and lets the batch endpoint come back quickly even on a warm Brisbane afternoon when the air-conditioning has pushed the data centre ambient up a few degrees. The set function on the batch object accepts each operation array, and a chain of method calls stitches everything together before you hand the batch back to the form API.

The callback itself takes the arguments you declared plus the standard batch context array. Inside that callback you load the entities referenced by the slice, apply your transformation, and let the function return. Drupal takes care of state storage between slices, so you do not need to write anything to the database yourself. A common idiom is to call entity_save or apply a service from the migration plugin suite, then push a count into $context['results'] so the finished callback can summarise the work later.

Showing progress and catching errors

Progress callbacks turn the batch into something a non-technical user can trust. The signature is straightforward: receive the batch context by reference, an integer for the total, an integer for current progress, and a label. Read $context['sandbox']['progress'] and $context['sandbox']['max'] if you are writing your own sandbox counters, or set the message string in $context['message'] so Drupal renders it under the progress bar. For long-running Australian projects we usually pair this with a watchdog entry at each milestone so the team back in Canberra has an audit trail.

The need for resilient batching is not limited to enterprise builds. High-volume content sites like worldwidemom's archive, which juggles thousands of archived posts, run into the same pressure points when they switch stacks or backfill categories. The same operational habits that protect a university migration protect a long-running personal archive.

Error handling is where the Batch API tends to surprise first-time users. By default, if a single operation throws an exception, the whole batch stops and the user sees a generic error page. For production jobs we recommend catching exceptions per item, logging them through the Drupal logging service, and continuing with the rest of the slice. That way one corrupt node does not derail a migration of forty thousand records.

It is also worth thinking about idempotency. A batch that crashes halfway through will be replayed when the user clicks resume, and any operation that is not idempotent could end up doing the same work twice. Use unique identifiers on external systems, or at least check for a flag on each node before applying your change. This is one of the reasons local Drupal teams serving finance and health clients in Sydney tend to lean on the Migrate API alongside Batch rather than rolling their own loop.

Running batch work from Drush

Form-based batch jobs are great for interactive use, but the moment you need to run a cleanup on a server overnight, Drush is the friendlier interface. The trick is to build the same batch array inside a Drush command, then pass it to drush_batch_command and call drush_batch_process. Drupal treats the resulting process the same way as a browser-initiated batch, so all of your callbacks continue to work. There is a walkthrough of writing that command in the Drupalwoo notes on Drush batch commands that gets the details right.

AEST and AEDT scheduling matters more than people expect. Drush picks up cron timing from the system clock, so if your batch is meant to run at three in the morning in Sydney but the server is set to UTC, you will trigger it ten hours earlier than planned. Keep the timezone in lockstep between the host and your scheduler, and double-check daylight saving dates around the first Sunday of October and the first Sunday of April. Australian Drupal shops have lost entire Saturdays to timezone slip-ups because of this.

Drush batches also behave better with a longer PHP execution time, since the command loop runs inside a single shell. Bump memory_limit and max_execution_time before launching the job, ideally through environment variables in your deployment script so the change is documented and reversible. A local cron job wrapped around the Drush command is the most common pattern we see on production stacks running Apache or Nginx in front of an FPM pool.

Performance tips and pitfalls

The biggest performance lever is the size of each slice. Process too few records per operation and the overhead of request restarting dominates; process too many and you hit PHP limits again. Most Australian teams settle between fifty and two hundred records per slice for entity save operations, dropping to twenty or so for jobs that involve remote API calls. The slowest step sets the pace, not the fastest.

Watch out for the silent killer of static caches. When a batch processes thousands of nodes, the entity static cache grows until PHP runs out of memory. Clear it at the start of each operation using drupal_static_reset, and your batch will hold a steady memory footprint across thousands of slices. Pair this with a fast entity type and you can push past 50,000 nodes on a modest VPS in Melbourne or Brisbane without breaking a sweat.

Finally, remember that the Batch API is a transient processor, not a long-term queue worker. If your job needs to survive a server restart or span multiple machines, look at the Queue API or the Ultimate Cron queue plugins instead. Many teams reach for Batch first and then rebuild the workflow later; a deliberate choice up front saves a rewrite. A batch that finishes in three minutes does not need a queue, but a batch that must be paused and resumed over a weekend often does.