Storage provider

A storage provider lets Runlane keep run state in a database. Core owns the records and workflow rules. The provider makes the database behave like StorageDriver.

Runlane core
    |
    | get, query, transact
    v
StorageDriver
    |
    v
your database

Import StorageDriver and its supporting contracts from @runlane/contracts.

Check the database first

The database must support the behavior Runlane needs:

• atomic changes across all records in one transition;
• version checks that stop stale writers;
• ordered index queries with stable pagination;
• bounded record and transaction sizes;
• one stable time value during a transaction attempt.

If the database cannot commit the full transition atomically, it cannot be a conforming storage provider. Do not split one transition into several commits.

Implement three operations

get() returns one position for every requested key, including repeated keys and missing records. Return copies so callers cannot mutate stored objects, dates, or bytes.

Make transactions safe to replay

withTransaction() carries most of the contract:

read records
     |
query index candidates
     |
call now() and seal reads
     |
stage create, replace, or delete
     |
validate versions and commit everything
     |
retry the whole callback after a supported conflict

The callback can run more than once. Do not call external APIs, run user code, generate random ids, or emit meaningful logs inside it.

Important rules:

• now() returns one frozen time for the current attempt.
• Reads stop after now() or the first mutation.
• create() succeeds only while the key is absent.
• replace() and delete() use the exact version read by that attempt.
• A failed condition rolls back the whole attempt.

The storage contract details page contains every transaction rule and edge case.

Optimize storage mechanics without moving workflow rules

Two optional transaction features let a provider reduce contention:

• append(command) allocates and persists an ordered batch of opaque values in the same commit as the point writes. Core uses it for observations; providers without it keep the existing point-write path.
• queryCandidates(command, { mode: StorageCandidateMode.Exclusive }) owns returned candidates until the attempt ends and skips busy candidates. Advertise profile.exclusiveCandidates: true only when the provider guarantees that behavior across instances.

Core continues to own claims, completion, retries, leases, and queue capacity. These features change storage mechanics without adding workflow methods or changing application APIs. Follow the ordered append and exclusive selection rules before enabling them.

The transaction is already a bounded write batch. create(), replace(), and delete() stage records; they do not require a database request per call. A provider can group those writes by collection and operation, validate all conditions, and commit them together. A failed condition must abort the entire attempt, even if some groups have already executed. Do not introduce provider-specific lifecycle commands to obtain bulk writes.

Core can combine independent worker lifecycle transitions into one transaction and combine concurrent point reads into one get() call. It respects the advertised limits and keeps completion pending until the transaction commits. Providers must handle complete batches correctly; they must not assume that a transaction concerns only one run. Existing providers without ordered append continue to use individual lifecycle transactions.

Treat index rows as candidates

Runlane owns a closed set of StorageIndexName values. Map each name to a native index; do not expose arbitrary provider queries through StorageDriver.

Sort by the encoded sort value, then the encoded record key. Use Runlane's cursor codec. A cursor is valid only for the same index, partition, range, and direction.

An eventually consistent index may return stale rows or miss recent rows. Core handles that by reading a useful candidate again by its exact key before making a write decision.

Store Runlane documents unchanged

Runlane sends a key, an opaque versioned JSON document, and its complete index entries. Use the public storage codecs for keys, tuples, dates, cursors, and unsigned integers.

Do not create another editable schema for runs, events, steps, schedules, leases, outbox messages, or wait tokens. Core owns those document shapes.

Report what the provider can guarantee

The storage profile declares:

• durability: process memory or durable storage;
• clock authority: client, server-sampled, or transaction-authoritative;
• index consistency: strong or eventual;
• point-read, query, record, and transaction limits.

Enforce the limits you report. A weaker supported profile is better than a stronger false claim.

Add operator reads only when needed

OperatorReadDriver provides exact run lookup, filtered run lists, and event history. It is optional. A lane can execute tasks without it.

If you add it, update its projections in the same commit as the raw records. Tombstoned runs and events must disappear from operator reads immediately.

A separate read contract does not require separate document storage. Keep canonical payloads once where practical, and derive only the metadata needed for indexed filtering and ordering. PostgreSQL stores that metadata beside its canonical run and event records and reads them directly with bounded SELECTs. Other providers may use different physical layouts while returning the same records and cursor behavior.

Plan the atomic boundary for each database

SQL providers can group statements inside a native transaction. The read-set validation must still protect missing keys and blind creates, not only rows returned by a locking read. MySQL's locking reads and SKIP LOCKED are mechanisms to evaluate against those guarantees; skipping busy candidates does not replace validation of the full transition.

Redis needs an atomic boundary covering the complete read set, records, indexes, and optional stream allocation. Redis transactions do not roll back commands that fail during execution. A pipeline or MULTI wrapper alone therefore does not prove Runlane's all-or-nothing contract. Validate the complete batch before mutation and test failure paths against the actual implementation. Runlane does not ship MySQL or Redis storage providers today.

Redis needs special care

Runlane may touch unrelated keys in one transaction. A normally sharded Redis Cluster cannot transact across those keys.

Putting every Runlane key in one hash slot can provide one atomic boundary. Document that as a single-slot setup, not a normally sharded Redis deployment. Runlane does not ship a Redis provider today.

Prove the provider

Run the shared storage conformance suite before composing the driver into a lane. Then test the real database under conflicts, failures, restarts, and shutdown.

Continue with test a provider and the exact storage values. Use the Postgres driver as the production reference. createLocalStorageDriver() from @runlane/local-adapters is the smaller executable reference.