Drupalwoo ☉

Processing Large Data Sets with Drupal Batch API

Large imports often begin with a CSV upload and end as a stalled request, a full disk, or duplicate catalogue records. Drupal's Batch API divides long-running PHP work into manageable steps while preserving progress between iterations. It suits node, taxonomy, user, and migration updates where handling everything inside one controller request would exceed PHP limits or block visitors.

The API supplies progress messages, a finished callback, failure messaging, and a recovery point for interrupted work. It can run behind a normal Drupal form while site owners watch counters advance. For continuous, high-volume workloads, a queue worker may be a better fit; batches are strongest for finite imports, scheduled transformations, and administrator-triggered maintenance.

Why batching matters for large Drupal workloads

A direct loop that creates thousands of entities ties database work, object construction, and validation to one request. Memory grows with the loaded result set, while a slow PHP or reverse-proxy limit can terminate the job before completion. Visitors may receive a timeout, and rerunning the request can create duplicate records unless every write is idempotent.

A Drupal batch replaces that single oversized request with repeated operations. Each callback can process a bounded number of records, update the sandbox with its position, and set a progress message. The batch framework persists its state, so an administrator can refresh the status screen or resume after a recoverable interruption rather than beginning again.

Batching improves capacity and observability, not raw throughput. Creating and saving one node at a time still carries a database cost per record. Measure realistic samples, tune the chunk size, and move only the work that genuinely needs background execution out of the web request.

Designing a restartable data workflow

Start with a clear data contract: define required fields, accepted formats, maximum lengths, and what constitutes a valid row. Keep the source file in a private managed-file location, and stage validated records before changing published content. This creates an auditable boundary between untrusted input and Drupal entities.

Split the job into stages such as validate, transform, write, and finalise. Store checkpoints as stable identifiers or offsets rather than relying on a PHP session. A stable source identifier, such as a supplier SKU or an external node UUID, also lets a retried stage recognise records it has already handled.

Make every stage safe to repeat. Upsert entities by source identifier, use transactions only around small related groups, and record rejected rows in a review file. If the job reaches its final stage, check that expected counts and required fields still match before publishing or sending downstream updates.

Implementing staged operations in Drupal

Start from a controller, form submission, or administrative command rather than from a front-end page. The injected Drupal\Core\Batch\Batch service works in Drupal 10 and 11. Configure the operations, finished callback, and error message, then call process() to hand control to the batch runner.

/** @var \Drupal\Core\Batch\Batch $batch */
$batch = \Drupal::service('batch');
$batch->setTitle(t('Importing product records'));
$batch->setOperations([
  ['import_product_rows', [$file_uri]],
]);
$batch->setFinishedCallback('import_product_rows_finished');
$batch->setErrorMessage(t('The product import could not be completed.'));
$batch->process();

The callback receives the file URI and a reference to the shared context. This example reads a manageable number of rows per iteration, saves the file offset, and uses that offset when the batch calls the operation again.

function import_product_rows($uri, &$context) {
  $context['sandbox']['offset'] ??= 0;
  $context['sandbox']['imported'] ??= 0;

  $path = \Drupal::service('file_system')->realpath($uri);
  if (!$path || !is_readable($path)) {
    throw new \RuntimeException('Cannot open the import file.');
  }

  $handle = fopen($path, 'r');
  fseek($handle, $context['sandbox']['offset']);

  $storage = \Drupal::entityTypeManager()->getStorage('node');
  $processed = 0;
  $total_bytes = max(1, (int) filesize($path));

  while (($row = fgetcsv($handle, 0, ',', '"', '\\')) !== false && $processed < 50) {
    if ($context['sandbox']['offset'] === 0 && $row[0] === 'sku') {
      $context['sandbox']['offset'] = ftell($handle);
      continue;
    }

    $node = $storage->create([
      'type' => 'product',
      'title' => $row[1],
      'field_sku' => $row[0],
    ]);
    $node->save();

    $context['sandbox']['offset'] = ftell($handle);
    $context['sandbox']['imported']++;
    $processed++;
  }

  $at_end = feof($handle);
  fclose($handle);

  $context['message'] = t('Imported @count products.', [
    '@count' => $context['sandbox']['imported'],
  ]);
  $context['finished'] = $at_end
    ? 1
    : min(0.99, $context['sandbox']['offset'] / $total_bytes);
}

Replace the example field names with those defined by the content type, and validate each row before saving it. The callback assumes a readable local managed file; remote storage adapters may require a different stream-reading method. A production implementation can register the importer as a service and use its callable method as the operation. The finished callback should report the final imported and rejected counts without exposing sensitive source data.

Comparing execution strategies

The batch API is request-aware and provides a friendly status screen, but it is not a continuously running worker. Choose the mechanism based on arrival pattern, expected volume, and recovery requirements. The following comparison shows where each approach fits.

Method Best fit Operator visibility Recovery model
Batch API Interactive imports and finite transformations Batch status and progress messages Stored sandbox and checkpoints
Queue workers Continuous or high-volume processing Queue metrics and worker logs Item-level retries and deadlines
Cron Scheduled maintenance and cleanup Cron logs and task history Rerun from a defined start point
Drush or custom CLI Deployments, migrations, and system administration Terminal output and log files Shell-controlled checkpoints

Use a batch when an editor needs visible progress and the job has a finite source. Use queues when records arrive continuously or processing must survive web-request limits. Cron is suitable for recurring maintenance, while a controlled command-line process often suits deployment-time migrations.

A hybrid can work well: a batch validates source data and creates queue items, workers write the entities, and another batch verifies counts before final publication. Whichever approach is chosen, prevent two workers from consuming the same work and make writes idempotent so retries do not create duplicates.

Protecting data and handling failures

Treat imports as untrusted input. Restrict uploads to authorised users, place files outside the public path, validate MIME type and extension, and check the real size. Use entity APIs and typed field values rather than concatenating values into database queries. If spreadsheet content will later be exported, neutralise formula prefixes as well.

Design failure handling before launch. Catch row-level validation errors, preserve the original row number and reason, and continue only when later records do not depend on the failed item. Avoid one transaction around a million-row import; it increases lock time and makes a small error roll back useful work. The finished callback receives a final success flag, while fatal exceptions should trigger the batch's failure message and a useful log entry.

Australian teams must also account for privacy and communications law. The Privacy Act 1988 and Australian Privacy Principles may apply when imported records contain personal information. The Spam Act 2003 sets consent, identification, and unsubscribe requirements for commercial electronic messages, while state and territory privacy rules can add obligations. Minimise collected fields, restrict access, set retention periods, and keep personal data out of routine logs and status messages.

Monitoring and integrating batch workflows

Give each run a correlation identifier and record its source file, operation version, start time, processed count, failure count, and final outcome. Monitor runtime, memory use, database contention, and processed-row rate against a representative sample. If a batch slows sharply, reduce its chunk size or move writes to a queue before workers begin timing out.

When imported entities feed a headless front end, finish and validate the Drupal entities before exposing them. A practical guide to custom content types shows how records can be exposed through JSON:API while preserving the relationships and access controls that matter for API consumers.

Teams spanning Sydney, Melbourne, and Brisbane should store timestamps in UTC and display them in the relevant state time zone, including AEDT where daylight saving applies. Supplier catalogue changes commonly arrive as CSV spreadsheets, making a small sample and count reconciliation a normal part of Australian administration workflows. For local retail catalogues, normalise prices to Australian dollars and record GST treatment explicitly.

Before starting

During processing