Storage contract details

Use this page while implementing or reviewing a StorageDriver. Start with build a storage provider for the shorter guide.

Keep logical records in core

NewStorageRecord contains a logical key, a JSON document, and all index entries. The provider assigns a new opaque positive-decimal StorageRecordVersion only when a mutation commits.

Every current logical document has this envelope:

{ "schemaVersion": 1, "value": {} }

Only schema version 1 is accepted. There is no compatibility reader for another version. A malformed value or a mismatch between the document, key, and derived indexes is durable corruption.

Keep documents opaque unless the provider has a contract-aware projection. Use the exported codecs and table definitions. Do not maintain another schema for runs, events, steps, outbox messages, schedules, owners, queue capacity, observations, leases, tombstones, or wait tokens.

JSON boundaries reject non-finite numbers and reference cycles during parsing and encoding. Parsed records and materialized appends own their nested JSON values. Reuse those owned values within the boundary instead of parsing or copying them again; new caller inputs and returned provider data still require validation at their respective boundaries.

encodeStorageDocument(codec, value) applies the owning document codec and produces an immutable canonical JSON document. parseNewStorageRecord() returns an immutable record, and parseAppendRecordsCommand() owns immutable appended values. Treat these write inputs as readonly. The helpers can reuse snapshots they previously validated and froze, including their UTF-8 byte measurements. Freezing an arbitrary caller value does not establish validation. Mutable caller inputs are reparsed, and ordinary stored-record reads retain independent caller-owned copies.

This reuse is an in-process optimization tied to the snapshot's lifetime; durable state and transaction fences remain in storage. Providers continue to enforce their advertised limits with the shared measurement helpers. No provider-specific trust flag or additional transaction method is required.

The codecs also check identities that JSON shapes cannot express alone. These include:

• deterministic step tokens and schedule occurrence ids;
• nested environment and run ownership;
• observation source identity and unsigned 64-bit positions;
• complete outbox claim state;
• wait-token resolution, link counters, and reciprocal idempotency ownership.

Preserve completion-wait presence

RunCompletionWaitPresence is a durable, monotonic scan-elision hint. Before core can persist a run waiting for a source run to finish, it creates this source-id marker unless the same fenced read already proves that the source is terminal. An absent marker therefore proves that no completion wait was registered for that source id, so a terminal transition can skip the waiter-index query. A present marker is only a conservative signal to query RunsWaitingForCompletion; that named index remains the authoritative waiter set.

The marker uses ordinary point reads and conditional creation. It works across processes and provider profiles without a new driver method. Providers must store it like every other logical collection. Do not derive it from an in-memory cache, delete it during run pruning, or interpret it as a waiter count. Ordinary runs create no marker. One small row remains for every source id ever used by a completion wait, including a future id whose source is never created. Safe reclamation requires an atomic or reference-counted wait-edge protocol.

This hint does not repair the existing crash window between committing a terminal run and scanning or releasing its waiters. A present marker on an eventually consistent named index also retains that index's existing possibility of temporarily omitting a newly registered waiter. A missing or unavailable terminal-transition hint must query the canonical index conservatively.

Keep operator reads separate

OperatorReadDriver is an optional read model with getRun(), listRuns(), and listRunEvents(). A driver can execute tasks without it. Core can fall back to the typed raw-record path for one exact run lookup, but list and history queries require the read model.

When a provider adds a projection, update it in the same atomic commit as the raw records. A visible run must appear with its exact summary and event history. A tombstoned run and its events must disappear in that same commit. Raw child cleanup may continue later.

The read-model boundary does not prescribe a second physical document copy. A provider may keep indexed metadata beside, or linked to, the canonical records. A joined read must return one consistent view of the metadata and documents, remain bounded, and preserve the same filtering and cursor semantics. Projections are derived from canonical records and are not independently writable state.

Bind every filter and sort input into the cursor scope. Return defensive copies of run records, summaries, events, nested values, and dates.

An opaque document that does not decode as a canonical run or event remains a valid raw record. Do not guess at its fields or add it to the operator projection. Core's typed table boundary owns logical validation.

Treat each transaction callback as a new attempt

A provider may replay the callback after a supported conflict. Only the committed attempt's return value may escape.

1. get() reads records or confirms that they are absent. Both results join the validated read set.
2. queryCandidates() returns index references, optionally holding exclusive ownership. Core point-reads every candidate that affects a decision.
3. The first now() call seals the read set and returns one frozen time for that attempt.
4. create(), replace(), delete(), and optional append() stage mutations.
5. The provider validates the read set and commits all staged mutations atomically.

Reject get() and queryCandidates() after now() or the first mutation. Reject overlapping reads. Roll back the attempt if the callback throws or a staged condition loses a race.

create() is a conditional insert. It may target an unread key or a key read as absent. It must reject a key read as present, and commit only while the key remains absent.

replace() and delete() require the unchanged record and version returned by the same attempt. A key may be mutated at most once. A replacement cannot change the key. Replacing a record replaces all index entries in the same commit. Deleting it removes them.

Providers may group staged mutations into native bulk statements without adding another batch method. Grouping does not change per-record conditions: all expected creates, replacements, and deletions must succeed, and any mismatch rolls back every group, projection, and append in the attempt. Never report a partially applied batch as a successful transaction.

Do not run user code, network calls, random id generation, or externally meaningful logging inside the callback. Prepare stable identities before entering it.

Add optional ordered append

StorageTransaction.append(command) stages one ordered batch of opaque JSON objects. The command selects a StorageStreamName, a partition, and a non-empty values array. Storage assigns positions during commit; input values must omit the stream's position field. The method returns no allocated positions.

Core owns observation contents and lifecycle rules. Providers translate the canonical storageStreamLayouts and document codecs. materializeStorageAppend() defines the persisted head, records, keys, and indexes for a starting position. readStorageStreamPosition() validates the existing head; an absent head starts at position 1. Corrupt or exhausted heads must fail without changing durable state.

For StorageStreamName.Observations, the partition is the environment name. Positions are positive unsigned 64-bit decimal strings. Every batch is contiguous and follows previously committed batches in that partition. The next-position head must also fit the unsigned 64-bit range. Do not convert positions to JavaScript numbers.

The allocator, appended records, point mutations, and projections commit atomically. Aborted or replayed attempts consume no stream positions and leave no records. Coordinate through durable storage across provider instances and processes. Native append and existing point writers use the same head and record layout; protect absent-record dependencies as well as present versions.

An attempt may stage only one append batch. It cannot also point-read or mutate the appended stream's head or records in the same partition. Appending ends the read phase just like a point mutation. Copy the supplied values at the boundary and reject use after the attempt ends.

The head and every appended record count toward transaction record and byte limits, together with point dependencies and mutations. measureStorageAppend() accounts for the maximum position width before allocation and checks per-record size. Reject oversized atomic work; never split it into commits.

Providers may omit append. Core then uses the existing head read and conditional point mutations. Application APIs and persisted documents are the same on either path.

Add optional exclusive candidates

A provider may advertise profile.exclusiveCandidates: true and accept queryCandidates(command, { mode: StorageCandidateMode.Exclusive }). Omitted mode, Advisory, and ordinary query() retain advisory behavior.

Exclusive selection skips records held by another exclusive attempt and holds every returned record until the current attempt commits or rolls back. Ownership must work across provider instances and processes. Core still point-reads selected records before making decisions; selection does not replace version checks, lease fences, or capacity rules.

Lock only returned records. Do not lock an extra lookahead row to determine pagination. A full exclusive page may return a cursor even if the next page is empty. Rows skipped because they were busy may require a fresh scan after their owner finishes; an empty exclusive page is not proof that the index is empty.

Do not advertise this capability if selection remains advisory. Core requests exclusive mode only from providers that advertise it.

Use the closed index catalog

query() and queryCandidates() accept only StorageIndexName values. Map each value to a native query plan. Do not expose a generic predicate or scan API through StorageDriver.

Index results contain only { key, sort }. Order them by encoded sort, then encoded record key. Treat every result as a candidate because a provider may report eventual index consistency.

Use the public cursor functions:

• encodeStorageQueryCursor() to return a continuation;
• decodeStorageQueryCursor() before applying one;
• parseQueryRecordsResult() in boundary tests.

A cursor binds its schema version, index, partition, range, direction, last sort, and last record key. Reject it when any part of the query scope differs.

Use the canonical encodings

Use storageTupleCodec for tuple components and storageRecordKeyCodec for keys stored in maps or native provider representations. Compare two validated key objects with haveEqualStorageRecordKeys() instead of encoding both. The encoding is lowercase hexadecimal UTF-8 with / between components. Its lexical order matches component-wise UTF-8 byte order.

Use compareStorageEncodedValues() for the same comparison in memory. Use isoDateCodec for ordered timestamps and storageUnsignedIntegerCodec for unsigned 64-bit positions.

For an inclusive tuple-prefix range, call encodeStorageTuplePrefixUpperBound() and use its exclusive result. Do not append a Unicode sentinel. There is no maximum arbitrary Unicode string, so a value starting with U+10FFFF can still have more bytes.

Preserve pruning invariants

Run pruning does not trust the size of an advisory index. A visible run stores exact counts for events, outbox messages, steps, and wait-token links. Tombstoning copies those counts. It also removes the run from operational reads in one transaction.

Cleanup then works in bounded transactions. Each pass point-reads child candidates, deletes exact step-token owners and reciprocal token links, and reduces durable counters. The run and tombstone disappear only after every counter reaches zero. An empty eventual-index page is not proof that cleanup finished.

Wait-token pruning is separate. It point-reads a terminal candidate, keeps it while any run link remains, and deletes the token with its exact idempotency owner. A missing or non-reciprocal owner is an invariant failure, not permission to orphan either record.

Report a profile that is true

The profile describes mechanics:

• durability is process or durable;
• clock authority is client, server_sampled, or transaction_authoritative;
• index consistency is strong or eventual;
• optional exclusiveCandidates declares transaction-owned candidate selection;
• limits cover point reads, query pages, record bytes, transaction records, and transaction bytes.

Enforce every advertised limit at the provider boundary. Reject an oversized atomic transition with createStorageLimitError() so core can split a combined batch safely after rollback. Never split it into partial commits.

The limit values must work together. maxTransactionBytes must be at least maxRecordBytes. It must also be reachable within maxTransactionRecords records of at most maxRecordBytes each.

Use transaction_authoritative only when the full point-read set is sealed or locked before sampling a shared advancing clock. With client or server_sampled time, version and ownership fences still protect writes. Clock skew can still cause early duplicate execution or brief capacity oversubscription. Document that risk.

Use Local as the executable reference

createLocalStorageDriver() from @runlane/local-adapters uses immutable attempt snapshots, optimistic read-set validation, copy-on-write ordered indexes, non-repeating versions, and a client clock.

Its default profile is process-local, strongly indexed, and limited to 100 point reads, 100 query results, 100 transaction records, 400 KiB per record, and 4 MiB per transaction. maxTransactionAttempts defaults to 16. Tests may inject a clock or a complete limits profile.

This driver is a reference and development implementation. It is not durable production storage.

Know the Redis Cluster limit

A general sharded Redis Cluster cannot satisfy a transaction that touches arbitrary Runlane keys because a Redis multi-key transaction or script must stay in one hash slot.

Routing every Runlane key to one slot can satisfy atomicity, but it gives up normal cluster sharding. Describe that deployment as a single-slot topology.

Verification checklist

Before publishing a driver, prove:

• exact point-read positions, including duplicates and misses;
• defensive copies for objects, dates, and bytes;
• conditional create, exact replace/delete, atomic rollback, and callback replay;
• conflict detection for present, absent, read-only, and blind-create dependencies;
• frozen attempt time and rejection of reads after sealing;
• ascending and descending order, every range bound, tie-breaking, and cursor-scope rejection;
• atomic index movement and explicit limit failures;
• immediate tombstone visibility and operator-query parity when OperatorReadDriver is present;
• wait-token completion, timeout, reciprocal links, pruning, and all token index mappings;
• injected failure paths with no partial writes.

When implemented, also prove atomic ordered append, mixed-writer compatibility, position bounds, and disjoint exclusive selections across independent instances. Enable the corresponding optional conformance suites explicitly.

Use test a provider for the conformance harness and storage contract values for the closed names.