Telemetry reference

This page describes the runtime signals emitted through RunlaneTelemetry and the durable observation exporter. Use export traces, metrics, and logs for setup and export durable observations for checkpointed delivery.

Public entrypoints

The OpenTelemetry telemetry adapter accepts optional meter, tracer, and logger instances. Omitted instances come from the global OpenTelemetry APIs using instrumentation scope @runlane/observability. Context capture and extraction use the global context manager and propagator even when a tracer instance is supplied. The adapter does not initialize or shut down SDK providers.

Metrics

The table is a projection of runlaneMetricDefinitions from @runlane/contracts. It lists provider-neutral attribute keys and units. The OpenTelemetry adapter prefixes attribute keys with runlane. and replaces underscores with dots: task_id becomes runlane.task.id. It exports count units as 1 and preserves ms.

Metrics emit when the corresponding operation occurs; enabling an adapter does not pre-populate every series. Exporter metrics require telemetry on the exporter itself. Maintenance, outbox, and wait-token metrics require those operations to run. Millisecond duration metrics include failed attempts; the outcome/status attributes distinguish them.

runlane.runnable.scan.count counts completed queue-partition page reads in storage-polling claims, including empty pages. runlane.runnable.scan.candidate.count counts returned candidate references before merging or rejecting stale or capacity-blocked entries. A multi-queue scan counts each queried partition; fetched candidates are not necessarily claimed runs. Discarded transaction retries and failed transactions contribute neither counter. Transport execution that already knows the run id does not perform a runnable scan.

Attribute validation uses the allowed keys above. Run ids, trace ids, and payload fields are not metric dimensions. The adapter bounds keys, not the number of distinct application-defined queue, task, or environment names.

Traces

RunlaneSpanName defines these operation spans:

runlane.delivery.execute
runlane.maintenance.phase
runlane.observation.export
runlane.outbox.flush
runlane.release.resume
runlane.run.attempt
runlane.run.claim
runlane.task.handler
runlane.task.step
runlane.trigger
runlane.worker.execute_next
runlane.wait.token.complete
runlane.wait.token.create
runlane.wait.token.prune
runlane.wait.token.read
runlane.wait.token.resume

A queued trace ordinarily connects request → trigger → attempt → handler → step. Child triggers capture their current execution context. Retries and resumes reuse the stored run carrier. An explicit traceCarrier overrides capture; custom tracers without captureContext() retain manual propagation and child fallback behavior.

traceCarrierSchema permits string header names and values up to contractDefaults.traceCarrier.maxBytes in UTF-8 JSON, including keys and escaping. This schema validates explicit inputs, captured context, and durable carriers. Invalid or oversized capture reports adapter_contract_violation through a propagation telemetry error and returns no carrier. Invalid explicit trigger options reject with configuration_invalid.

Spans describe individual operations and attempts, not one continuously open span for the entire durable run. Error records contain sanitized error type, Runlane code, and retryability when known. Raw exception messages and stacks are excluded.

Runtime logs

RunlaneLogEventType defines these events. The OpenTelemetry log body and event name add the runlane. prefix:

adapter_contract_violation
capability_missing
delivery_ignored
lease_refresh_failed
lease_refresh_lost_ownership
lease_refresh_observed_cancellation
maintenance_lease_claim_failed
maintenance_lease_refresh_failed
maintenance_lease_release_failed
observation_export_failed
outbox_publish_batch_failed
outbox_publish_partial_failure
service_observer_failed
service_pass_failed
wait_token_eager_resume_failed
wait_token_invariant_repaired
worker_claim_lost
worker_execution_abandoned
worker_loop_failed

Logs include their level, time when supplied, environment, sanitized error code, and event-specific attributes. They describe operational events rather than every persisted lifecycle transition. Lifecycle facts are available as observations.

onTelemetryError receives synchronous adapter failures with a log, metric, span, or propagation event. Span and propagation failure metadata exclude trace carriers. This hook is best-effort and does not report asynchronous SDK export failures.

Runtime observation storage

Set observability.durable to true on every producer and worker runtime for an environment when a checkpointed observation exporter must consume its lifecycle facts. It defaults to contractDefaults.observability.durable, which is false. With the default, canonical run events, steps, and wait tokens still persist, and live observers and telemetry still emit; Runlane skips the additional observation rows and shared stream-head update.

Observation exporter options

All polling and retry delays must fit a Node.js timer. start({ signal? }) and exportOnce({ signal? }) are mutually exclusive on one exporter. close() is permanent and idempotent; it drains the active export and checkpoint, not the entire backlog. getStats() returns a copy of the current counters and last checkpoint cursor.