Run maintenance

Every production deployment needs a clear maintenance owner. Workers execute task attempts. Maintenance moves work that is due, deferred, or interrupted back toward execution.

Run one bounded pass

Use a scheduled function, cron job, or supervised command. Put the bounded call in that entry point, such as src/handlers/runlane-maintenance.ts:

import { type RunlaneRuntime } from '@runlane/core'

export async function runMaintenancePass(runlane: RunlaneRuntime) {
  await runlane.start()

  try {
    return await runlane.runMaintenanceOnce()
  } finally {
    await runlane.close()
  }
}

Pass the shared runtime from your application bootstrap. The explicit start() performs lane startup checks before maintenance reads or writes state.

One pass can:

• create runs for due schedules;
• request delivery for due retries and waits;
• time out due tokens and resume runs linked to terminal tokens;
• finalize expired attempts and cancellations;
• recover delivery and publish pending outbox messages;
• continue cleanup for pruned runs.

The result contains the records changed by that pass so you can publish operational metrics.

Set batch limits when one invocation needs a fixed budget. waitTokenTimeoutLimit and waitTokenResumeLimit both default to 100.

The result reports waitTokensTimedOut and waitTokenRunsResumed. Keep limits consistent across replicas unless their capacity differs intentionally.

MaintenancePhase.WaitTokenResolution owns timeout and resume passes. Token completion also tries one small resume pass for lower latency.

Completion commits first, so a resume failure cannot undo it. Maintenance remains the recovery owner.

Understand duplicate delivery

Several maintenance replicas can publish the outbox safely. Core finds one bounded page, claims each message with a fenced token based on storage time, and records publish results atomically.

Another replica can claim an outbox item after the old claim expires. The old publisher cannot write a stale success over the new claim.

An eager transport trigger can persist its initial outbox item already claimed to save a storage round trip. If that producer stops after the commit, recovery waits for contractDefaults.lease.duration, currently 5m, before another replica can claim the item.

The transport must still tolerate a duplicate when publish succeeds but its stored acknowledgement does not.

Run supervised service loops

For an always-on deployment, put the service loop in a separate process entry point such as src/maintenance.ts:

import { type RunlaneRuntime } from '@runlane/core'

export async function runMaintenanceServices(runlane: RunlaneRuntime) {
  await runlane.start()

  const services = runlane.startServices({
    onServiceError(error, { phase }) {
      process.stderr.write(`${phase}: ${error.message}\n`)
    },
  })
  const close = () => void services.close()
  process.once('SIGTERM', close)

  try {
    await services.waitUntilClosed()
  } finally {
    process.off('SIGTERM', close)
    await services.close()
    await runlane.close()
  }
}

The example writes service failures to standard error and removes its signal listener during shutdown. Replace that write with your production logger and alerting path.

Postgres-backed storage coordinates replicas with durable maintenance leases. This allows failover without making every replica run every pass. waitUntilClosed() observes the loops; close() stops them during shutdown.

Storage supplies lease time after the ownership row is read. Process-clock skew cannot steal or shorten a live lease.

If refresh fails, that replica stops the band. After expiry, another replica can take over. The old lease token cannot change the new owner's lease.

Monitor progress, not only process health

Alert on:

• repeated maintenance errors;
• growing pending outbox work;
• retrying or released runs past their due time;
• pending tokens past their timeout;
• terminal tokens whose waiting-run count does not fall;
• expired active leases or runs stuck in cancellation_requested;
• run tombstones whose remaining child count does not fall.

A healthy process heartbeat does not prove that each phase is progressing.

In an isolated environment, test one case from each phase you use: a due schedule, retry or release, token timeout and completion, expired lease or cancellation, and pending outbox message. Confirm another replica takes over after the lease owner stops.

Do not catch and discard storage failures. The scheduler or service error observer must see a failed pass so supervision can retry it.