fidzu : Drupal

The Core Culprit: Synchronous Key-Value Upserts

In a standard deployment, Drupal manages temporary data, system variables, and specific configurations through the State API. By default, this API utilizes the database-backed Key-Value store.

The architectural vulnerability lives directly inside \Drupal\Core\KeyValueStore\DatabaseStorage::doSet(), which handles the underlying database execution for the public set() method.

// core/lib/Drupal/Core/KeyValueStore/DatabaseStorage.php

protected function doSet($key, $value) {
  $this->connection->upsert($this->table)
    ->key(['collection', 'name'])
    ->fields([
      'collection' => $this->collection,
      'name' => $key,
      'value' => $this->serializer->encode($value),
    ])
    ->execute();
}

Every time \Drupal::state()->set() is called, it triggers a synchronous Upsert query. Under the hood, MySQL's InnoDB engine enforces row-level locks (via INSERT ... ON DUPLICATE KEY UPDATE mechanics) to maintain ACID compliance during these operations. Because the default architecture blocks the PHP thread until the database confirms the write, a severe queue forms at the database layer during concurrency spikes.

The Empirical Impact (Drupal 11 & 12 Architecture)

Architectural Note: Because the synchronous DatabaseStorage::set() mechanism and InnoDB locking architecture remain unchanged in the Drupal 12.x branch, this Drupal 11 benchmark serves as a 1:1 representation of Drupal 12's concurrency limitations.

To isolate and prove this behavior, we benchmarked a standard Drupal application handling a high-volume transactional endpoint.

• Testing Tool: Apache JMeter.
• Load Profile: 200 concurrent threads executing heavy writes simultaneously.
• Operation: A synchronous \Drupal::state()->set() wrapped in a database transaction to simulate a complex payload save.

The MySQL process list immediately revealed the bottleneck. Multiple independent database connections stacked up in the process list, all attempting to execute the exact same query on the jmeter_lock_test row simultaneously. Because the first thread locked the row to perform its save, all subsequent threads were trapped waiting for it to release.

This database queue directly correlates to a severe degradation in application performance. The application's throughput plummeted to just 4.6 requests per second. Threads queued relentlessly behind the locked row, pushing average application response times to a massive 33,310ms (33.3 seconds), with peak latency hitting 40,181ms.

Ultimately, the database connection pool saturated and began throwing lock wait timeout exceptions. The application buckled, resulting in a 29.99% failure rate where nearly a third of all transactions were dropped entirely.

The Enterprise Mitigation: Asynchronous Decoupling

To scale past this limitation and guarantee 99.99% uptime, the architecture must transition from synchronous database dependence to an asynchronous state model. In our high-concurrency deployments, we mitigate this bottleneck by decoupling the State API from MySQL entirely and offloading it to a highly available Redis cluster, combined with asynchronous message queuing for relational entity writes.

1. In-Memory State Offloading

By overriding the core state service in the application's services.yml, we reroute these synchronous database calls into memory. Redis handles tens of thousands of concurrent atomic operations per second without the locking overhead of a relational database.

# services.yml
services:
  # Override the default database-backed state service
  state:
    class: Drupal\redis\State\State
    arguments: ['@keyvalue', '@redis.client.factory', '@redis.settings']

2. Asynchronous Message Queueing

For mission-critical entity writes (e.g., bid submissions, order checkouts), we completely remove the write operation from the primary PHP thread. Instead, we serialize the payload and push it to a RabbitMQ broker via a custom service.

// High-Concurrency Transaction Controller
public function submitTransaction(Request $request) {
  $payload = $request->getContent();
  
  // 1. Acknowledge receipt instantly (0ms database lock)
  $this->rabbitMqService->publish('transaction_queue', $payload);
  
  // 2. Return 202 Accepted to the user immediately
  return new Response('Transaction queued successfully.', 202);
}

The Core Patch Proposal: Implementing DestructableInterface

While external infrastructure like Redis is standard for enterprise deployments, Drupal core should natively implement a Deferred Write-Behind Strategy within the State API to protect all environments out of the box.

By analyzing the production \Drupal\Core\State\State class (which extends CacheCollector), we can surgically remove the synchronous database bottleneck. Rather than executing $this->keyValueStore->set() immediately, the class should queue the updates in memory, allow the CacheCollector to handle the static request caching, and defer the actual database upsert() to the kernel termination phase using Drupal's native DestructableInterface.

// 1. Add the interface
use Drupal\Core\DestructableInterface;

class State extends CacheCollector implements StateInterface, DestructableInterface {

  // 2. Add the queue property
  protected array $deferredWrites = [];

  // 3. Update set() to queue the write rather than blocking the thread
  public function set($key, $value) {
    if (isset(self::$deprecatedState[$key])) {
      @trigger_error(self::$deprecatedState[$key]['message'], E_USER_DEPRECATED);
      $key = self::$deprecatedState[$key]['replacement'];
    }
    
    $this->registerKeySetDuringRequest($key, $value, parent::get($key));
    
    // Queue the database write instead of executing it synchronously
    $this->deferredWrites[$key] = $value;
    
    // Allow CacheCollector to manage the static request cache
    parent::set($key, $value);
    $this->persist($key);
    static::updateCache();
  }

  // 4. Update setMultiple() to follow the same queuing logic
  public function setMultiple(array $data) {
    foreach ($data as $key => $value) {
      $this->deferredWrites[$key] = $value;
      $this->registerKeySetDuringRequest($key, $value, parent::get($key));
      parent::set($key, $value);
      $this->persist($key);
    }
  }

  // 5. Execute the bulk database write after the HTTP response is sent
  public function destruct() {
    if (!empty($this->deferredWrites)) {
      $this->keyValueStore->setMultiple($this->deferredWrites);
      $this->deferredWrites = [];
    }
  }
}

Architectural Trade-off: Eventual Consistency
Moving writes to the KernelEvents::TERMINATE phase shifts the architecture from strict ACID consistency to Eventual Consistency. By removing the InnoDB row lock, concurrent writes rely on a "Last Write Wins" resolution at the end of the request lifecycle. Furthermore, if a PHP fatal error or server crash occurs before the termination phase, the in-memory state update will be lost despite the user receiving a successful HTTP response. While this trade-off is catastrophic for financial ledgers or strict relational entities, it is the exact right compromise for the State API, which primarily manages non-critical, self-correcting metadata.

By adopting this patch, Drupal core mathematically eliminates state-based database locking during the HTTP request. It consolidates multiple scattered state updates into a single, highly efficient setMultiple() bulk write executing safely in the background, yielding instant zero-lock latency for the end user.

The Bridge Mitigation: A Drop-in Custom Module

Until a write-behind strategy is adopted in core, engineering teams can patch this bottleneck today without requiring complex external infrastructure. By writing a lean custom module (deferred_state), we can decorate the core state service and implement the DestructableInterface ourselves, bypassing the relational lock entirely.

1. The Service Decoration (deferred_state.services.yml)

services:
  deferred_state.state:
    class: Drupal\deferred_state\State\DeferredState
    decorates: state
    arguments: ['@keyvalue']
    tags:
      - { name: needs_destruction }

2. The Asynchronous Class (src/State/DeferredState.php)

namespace Drupal\deferred_state\State;

use Drupal\Core\State\State;
use Drupal\Core\DestructableInterface;

class DeferredState extends State implements DestructableInterface {
  
  protected array $deferredWrites = [];

  public function set($key, $value) {
    // 1. Maintain CacheCollector request tracking
    $this->registerKeySetDuringRequest($key, $value, $this->get($key));
    
    // 2. Queue the write instead of calling $this->keyValueStore->set()
    $this->deferredWrites[$key] = $value;
    
    // 3. Update the underlying CacheCollector storage directly
    $this->storage[$key] = $value;
    $this->persist($key);
  }

  public function destruct() {
    if (!empty($this->deferredWrites)) {
      $this->keyValueStore->setMultiple($this->deferredWrites);
      $this->deferredWrites = [];
    }
  }
}

The Engineering Standard

Relying on page caching to scale a platform only masks the symptoms of read-heavy technical debt. True enterprise stability requires engineering the underlying infrastructure to absorb massive, concurrent writes without latency. By isolating the exact code block responsible for database locking and actively decoupling it via deferred termination events or asynchronous messaging, we mathematically eliminate the locking cascade and ensure the technology stack acts as a commercial asset, not a liability.

Request an Architectural Audit