Export durable observations

Use @runlane/observability to export sanitized durable facts independently of task execution. The exporter scans the storage observation stream in bounded batches and advances a consumer checkpoint only after the sink accepts a batch.

Durable observation storage is opt-in because it adds a sanitized record for every observed fact and advances one shared stream head. Set observability: { durable: true } on every runtime that writes to the exported environment. The default is false; canonical run history, steps, and wait tokens remain durable, while live observers and runtime telemetry continue without the duplicate export records.

Use the same setting across all producer and worker replicas for an environment. An exporter can only read facts written by runtimes with durable observations enabled.

Your lane needs persistent storage for export to survive a process restart. A separate exporter process must connect to the same storage and environment as the workers. The local lane keeps observations only in memory.

Create an exporter

Install @runlane/observability, @opentelemetry/api@^1.9.1, and @opentelemetry/api-logs@~0.220.0. Configure a Collector or backend that accepts OTLP/HTTP logs, then create an exporter using your runtime's storage:

import { createRunlaneObservationExportSource, type RunlaneRuntime } from '@runlane/core'
import { createRunlaneObservationExporter, RunlaneObservationSinkOwnership } from '@runlane/observability'
import { otlpObservationSink } from '@runlane/observability/opentelemetry'

export function createObservationExport(runtime: RunlaneRuntime, logsEndpoint: string) {
  return createRunlaneObservationExporter({
    consumer: 'primary-observation-export',
    environment: runtime.environment,
    source: createRunlaneObservationExportSource(runtime.lane.storage),
    sink: otlpObservationSink({
      endpoint: logsEndpoint,
      resourceAttributes: { 'service.name': 'runlane-observations' },
    }),
    sinkOwnership: RunlaneObservationSinkOwnership.Exporter,
  })
}

Pass the complete logs URL, such as http://localhost:4318/v1/logs. The sink accepts optional headers for authentication and timeout as a Runlane duration. It owns a dedicated OTLP log exporter and does not require an application-global SDK. To adapt another dedicated OpenTelemetry LogRecordExporter, use openTelemetryObservationSink({ exporter }).

In your process supervisor, keep the promise returned by exporter.start() and handle its rejection. For a scheduled bounded job, use await exporter.exportOnce() and inspect its status: exported, idle, failed, or aborted. A failed tick returns its error; the polling loop retries retryable failures and rejects terminal structured errors.

Shut down gracefully

Call await exporter.close() before closing storage. It stops new scans, interrupts idle polling or retry backoff, waits for the current sink export and successful checkpoint, and closes the sink when sinkOwnership is Exporter. It leaves unprocessed backlog for the next exporter using the same consumer id.

Repeated close() calls return the same promise. Shutdown rejects if the active operation or owned sink cleanup fails. New start() and exportOnce() calls reject after shutdown. A sink or storage call that never settles can delay shutdown; configure provider timeouts.

The default ownership comes from contractDefaults.observability.exporter.sinkOwnership and is Application. Use this mode for a shared sink and close it yourself after every user stops. openTelemetryObservationSink.close() shuts down its underlying log exporter, so use a dedicated provider exporter or coordinate its lifetime with all its users. Observation export never closes storage or the application's telemetry SDK.

start({ signal }) still supports cancellation. Aborting a signal may interrupt a sink call and leave the batch for replay. It does not close the sink or permanently close the exporter. Prefer close() when you want the current export to finish.

Check delivery and replay

Run one active exporter for each consumer id and environment. Checkpoints use compare-and-set, but two exporters can both send a batch before one loses the checkpoint race. Use a different consumer id for an independent destination.

Delivery is at least once: a crash after sink acceptance but before checkpoint persistence can replay the same records. Deduplicate by observation record id when your destination requires it. OTLP log ingestion does not itself promise deduplication.

Trigger and execute a task, then inspect exporter.getStats(). After a successful batch, recordsExported and checkpointAdvances increase and lastCheckpointCursor is set. During a sink outage, failures increase while the checkpoint stays unchanged. Restart with the same consumer and environment to resume after its saved checkpoint.

Pass telemetry: openTelemetryRunlaneTelemetry() to the exporter options to record its batch, record, failure, checkpoint, and lag metrics through an initialized application SDK. Lag is measured when a batch succeeds; it is not an independent backlog age probe during an outage.

Exported data

Each durable record also carries a stable record id, source identity, recorded time, and opaque stream position. Payloads, task outputs, token outputs, metadata, completion actors, baggage, and arbitrary propagation headers are excluded. Run-event logs can join a trace through traceparent and tracestate; step and token observations carry no trace carrier.

Live observability.observers receive sanitized observations after the canonical transition persists, regardless of the durable setting. Their callbacks are awaited and can delay the caller. Failures go to onObserverError without rolling back the persisted fact. Use durable export when you need checkpointed delivery.