Deploy with Postgres and SQS

Choose this lane for SQS-triggered Lambda functions or long-running SQS consumers. Postgres remains the source of truth. Each SQS message only tells a worker which run to check.

Deployment shape

producer
   |
   v
Postgres (run + outbox)
   |
   v
publisher --> SQS --> worker
                      |
                      v
                   Postgres

maintenance --> due and recovery work

Runlane stores the run and its pending wakeup together in Postgres. A publisher sends the wakeup to SQS. The consumer then reads Postgres before it claims or executes the run.

Install the packages and bind queues

npm install @aws-sdk/client-sqs @runlane/core @runlane/lane-postgres-sqs @runlane/postgres-storage @runlane/transport-sqs zod

The files below use one practical layout:

src/
├── tasks/send-email.ts
├── runlane.ts
├── worker.ts                         # long-running consumer
└── handlers/
    ├── runlane-sqs.ts                # Lambda consumer
    └── runlane-maintenance.ts        # scheduled Lambda
scripts/
└── migrate-runlane.ts

Use either the Lambda consumer or the long-running consumer. Both are shown so you can match the deployment you already run.

Put task code in its own module

Create src/tasks/send-email.ts. Producers import the task definition for its schema and id. Only workers execute run() and therefore need EMAIL_API_URL.

import { task } from '@runlane/core'
import * as z from 'zod'

const envSchema = z.object({ EMAIL_API_URL: z.url() })

export const sendEmail = task({
  id: 'email.send',
  schema: z.object({ to: z.email() }),
  async run({ to }, context) {
    const { EMAIL_API_URL } = envSchema.parse(process.env)
    const response = await fetch(EMAIL_API_URL, {
      body: JSON.stringify({ to }),
      headers: { 'content-type': 'application/json' },
      method: 'POST',
      signal: context.signal,
    })
    if (!response.ok) throw new Error(`Email API returned ${response.status}`)
  },
})

Create one runtime module

Create src/runlane.ts. Every producer, delivery handler, worker, and maintenance process imports this module so they share the same environment, task catalog, queue definition, storage, and SQS binding.

Set DATABASE_URL and RUNLANE_SQS_QUEUE_URL before the module loads. Configure the AWS SDK region and credentials through your normal SQSClient environment or client options.

import { SQSClient } from '@aws-sdk/client-sqs'
import { createRunlane, queue } from '@runlane/core'
import { postgresSqsLane } from '@runlane/lane-postgres-sqs'
import { sqsQueue } from '@runlane/transport-sqs'
import * as z from 'zod'

import { sendEmail } from './tasks/send-email.js'

const env = z
  .object({
    DATABASE_URL: z.string().min(1),
    RUNLANE_SQS_QUEUE_URL: z.url(),
  })
  .parse(process.env)

export const emailQueue = queue({ name: 'email', default: true })

export const runlane = createRunlane({
  lane: postgresSqsLane({
    postgres: { connectionString: env.DATABASE_URL, schema: 'runlane' },
    sqs: {
      client: new SQSClient({}),
      queues: [sqsQueue(emailQueue, { queueUrl: env.RUNLANE_SQS_QUEUE_URL })],
    },
  }),
  queues: [emailQueue],
  tasks: { sendEmail },
})

The same emailQueue value appears in the runtime queue catalog and the SQS binding. A missing or mismatched binding stops startup instead of sending work to the wrong destination.

Run migrations during deployment

Create scripts/migrate-runlane.ts and run it as a release or deployment step before the new application starts. Do not call it from a request handler or worker startup.

import { applyPostgresStorageMigrations } from '@runlane/postgres-storage'
import * as z from 'zod'

const { DATABASE_URL } = z.object({ DATABASE_URL: z.string().min(1) }).parse(process.env)

await applyPostgresStorageMigrations({ connectionString: DATABASE_URL, schema: 'runlane' })

Deployment tooling must validate and apply the version-one baseline before startup. Starting Runlane does not inspect or change the schema. Runlane 0.4 intentionally replaces the schema shipped with 0.3; a 0.3 database cannot be upgraded in place. Use a fresh schema and see the Postgres migration instructions.

Handle Lambda deliveries

For SQS-triggered Lambda, create src/handlers/runlane-sqs.ts and configure that export as the function handler:

import { runlane } from '../runlane.js'

export const handler = runlane.createDeliveryHandler()

The handler has the lane's native Lambda event and result types. It starts the runtime on the first invocation and reuses it when the Lambda environment stays warm.

Framework, storage, parsing, or persistence failures leave the message available for SQS redrive. A stored task result such as failed, retrying, released, or cancelled means Runlane handled that delivery; it is not an SQS failure by itself.

Run a long-lived consumer

If you run a service instead of Lambda, create src/worker.ts and use it as that service's process entry point:

import { emailQueue, runlane } from './runlane.js'

const worker = await runlane.createDeliveryWorker({
  concurrency: 8,
  queues: [emailQueue.name],
})
const close = () => void worker.close()
process.once('SIGTERM', close)

try {
  await worker.waitUntilClosed()
} finally {
  process.off('SIGTERM', close)
  await worker.close()
  await runlane.close()
}

The entry point handles cooperative SIGTERM shutdown. Your process supervisor should restart it when waitUntilClosed() rejects.

Deploy compatible task catalogs

An SQS worker receives from a physical queue before it resolves the run's logical queue and task id. Every worker consuming a physical queue must therefore register every task that producers can route there. An older worker that receives an unknown task records a terminal, non-retryable TaskNotFound failure for that run and acknowledges the SQS message after storing that result.

Before a producer creates a task on an existing queue, upgrade or stop every worker consuming that queue. To let old and new workers overlap, give the new task version a new task id, a distinct logical queue, and a distinct physical SQS queue consumed only by the new catalog. For Lambda, map that SQS queue only to the new function version or alias. Changing only the logical queue name does not isolate work when old and new bindings share the same physical SQS queue.

Upgrade every runtime that can run the outbox phase, including maintenance processes, with the new SQS binding before producers use it. Outbox claims span the environment rather than one queue, so an old publisher can claim a new queue's pending message but cannot publish it through a missing binding. Keep old task definitions and both queue bindings available until queued, retrying, released, waiting, delayed, and redriven work has drained. If a task id remains unchanged, its handler and saved payload, task-output, and step-output shapes must remain compatible with runs created by the previous release.

Tune delivery capacity

Each limit belongs to one part of the system.

Use a standard SQS queue when tasks do not require broker ordering. Lambda fits bursty work that finishes comfortably within its execution limit. A long-running consumer fits sustained work and tasks that may outlive that limit.

The Postgres adapter's limits object sets record and byte limits for storage operations. It does not set pool or database capacity.

Size Lambda delivery

For a standard queue, Runlane starts every record in a Lambda event at the same time. Use this upper bound for full batches with Lambda's default polling and an event-source concurrency cap.

active deliveries =
  event-source maximum concurrency x batch size

Within one FIFO event, Runlane handles records in order and stops at the first record that cannot finish. Other message groups can run in separate Lambda invocations.

Lambda's Provisioned Mode cannot use event-source maximum concurrency. It scales between configured minimum and maximum poller counts. Each poller can support up to 10 concurrent invocations. Use maximum pollers x 10 as the invocation upper bound when sizing the database. Provisioned Mode costs more and is intended for queues that need faster, more consistent scaling.

When you use Provisioned Mode, AWS estimates poller demand with this formula.

events per second per poller =
  minimum of(
    round up(1,024 / average event size in KB),
    round up(10 / average function duration in seconds) x batch size,
    minimum of(100, 10 x batch size)
  )

required pollers =
  round up(
    peak events per second
    / events per second per poller
  )

AWS uses mean function time in this formula. Use p95 time for a safer workload plan.

Estimate the task capacity you need from measured traffic.

required active deliveries =
  peak deliveries per second
  x 95th-percentile (p95) handler duration in seconds
  x burst-headroom multiplier

required Lambda concurrency =
  round up(
    required active deliveries
    / observed records per invocation
  )

In the default polling mode, also calculate a starting cap. Available Lambda concurrency is what remains after other event sources and functions use their share.

starting Lambda invocation cap =
  minimum of(
    active-invocation database budget,
    downstream-safe Lambda concurrency,
    available Lambda concurrency
  )

required Lambda concurrency <= starting Lambda invocation cap

This is a planning limit, not proof that the deployment is safe. If required concurrency exceeds the starting cap, the deployment cannot meet the target rate with the current budgets. Increase database capacity or downstream quota, or use a different worker topology. Test the chosen cap and reduce it when database connections, latency, or downstream errors approach their limits.

Measure how full each batch is. Do not assume that every invocation contains the configured maximum. A busy queue may fill batches, while low traffic may produce one-record invocations.

If a downstream provider publishes request and token quotas, convert both to tasks per second. Use both quotas for a task that makes one provider request.

provider-limited tasks per second =
  minimum of(
    requests per minute / 60,
    tokens per minute / average tokens per task / 60
  )

Reduce the result when one task makes more than one request. Keep this limit separate from Lambda and database capacity.

For the SQS event source:

• Start with a batch size of 10. Larger batches start more work inside one Runlane handler. Test them under load first.
• Enable partial batch responses. This keeps one failed record from redelivering successful records.
• In default polling mode, reserve enough function concurrency for all of its event sources. The reservation must cover the sum of their concurrency caps.
• Set the queue visibility timeout using visibility timeout >= 6 x Lambda timeout + batching window. Set the dead-letter redrive count to at least 5.
• Move work that may approach Lambda's 15-minute execution limit to a long-running consumer.

These requirements come from AWS's current SQS event-source configuration, scaling, and partial batch response guidance. The AWS example's values belong to its measured database budget. They are not Runlane defaults.

Budget PostgreSQL connections

Each postgresSqsLane() instance creates its own Postgres driver and internal pg.Pool. The current driver cannot accept an external pool or configure the pool maximum. Each instance can therefore lazily open up to node-postgres's current default maximum of 10 connections. Add any application database pools in the same process to that number.

First estimate connection demand from active Lambda calls.

available consumer connections =
  database maximum
  - operational reserve
  - producer, observer, maintenance, and other fixed pool maximums

maximum connections per Lambda environment =
  Runlane pool maximum
  + application pool maximum
  + other pool maximums

active-invocation database budget =
  round down(
    available consumer connections
    / maximum connections per Lambda environment
  )

nominal connections for active Lambda invocations =
  event-source maximum concurrency
  x maximum connections per Lambda environment

known-process pool budget =
  operational reserve
  + sum(
      maximum live long-running processes for each deployment unit
      x maximum connections opened by each process
    )

known-process pool budget <= database maximum

For each process, add the Runlane pool and every application pool it owns. For Lambda, that sum is the maximum connections per environment. Include HTTP functions, maintenance functions, producers, and long-running worker replicas. The known-process formula is a hard configured bound only when every process count is bounded.

The Lambda formulas estimate active calls only. Lambda may freeze an environment and keep its pools open after a call ends. It may also replace an environment. Thus, an event-source cap does not set a hard limit on open database connections. Keep headroom and measure connections under sustained load.

RDS Proxy can reuse connections and set a firmer limit for the database backend. It does not cut Runlane's SQL or write-ahead log (WAL). Measure connection borrowing and session pinning before raising consumer concurrency.

For example, each Emails Lambda environment can open up to 10 Runlane connections. Its application pool can open 2 more. The function reservation and event-source cap are both 2. At two active invocations, the nominal pool budget is 2 x (10 + 2) = 24 connections before other services and the reserve. The measured stage still peaked at 27 after fixed services were included. Because Lambda may retain other warm environments, 24 is not a hard connection ceiling. These values protect that example's small database. They are not general defaults.

Estimate database work separately from pool limits.

connections needed for task SQL =
  tasks per second
  x measured connection-seconds per task
  x headroom multiplier

PostgreSQL requests per second =
  tasks per second x measured SQL calls per task

WAL bytes per second =
  tasks per second x WAL bytes per task

Measure connection-seconds as total pool checkout time divided by completed tasks. This also covers batches that share one connection. Do not count time spent waiting for an external API or model.

Choose database compute from the observed workload. Provisioned non-burstable capacity fits steady traffic. Aurora Serverless v2 fits traffic with large peaks and valleys. Reader replicas do not raise Runlane lifecycle throughput. Claims, leases, events, and outbox updates use the writer.

For Aurora, start with Standard storage for low or moderate I/O. AWS says I/O-Optimized can cost less when I/O exceeds 25% of total Aurora database spend. Use billed I/O and database metrics to decide. Runlane SQL counts are not Aurora I/O request counts.

Size long-running workers

The concurrency: 8 in the example above is illustrative. Set each worker's concurrency from handler duration, downstream limits, memory, CPU, and database latency. Add replicas when queue age or backlog grows too large. Keep their combined pools inside the database connection budget.

required worker slots =
  peak deliveries per second
  x p95 handler duration in seconds
  x burst-headroom multiplier

required worker processes =
  round up(
    required worker slots
    / measured safe slots per process
  )

steady-state target backlog =
  target average time in the selected backlog, in seconds
  x measured total departures per second

backlog growth per second =
  tasks entering the measured backlog per second
  - tasks leaving the measured backlog per second

This is a steady-state estimate. Choose one backlog population, such as runnable runs awaiting dispatch. Use the time spent in that population and the same population for both rates. If you count all accepted non-terminal runs, use total run time instead of queue wait. Departures include every transition that removes a task from the chosen population. This may include failure, cancellation, or dispatch.

Use queue age with backlog when scaling workers. Sustained positive backlog growth means the deployment is falling behind. A large backlog of short tasks may be healthy. A smaller backlog of old tasks may already miss the service goal. For a bounded Runlane queue, include runs waiting in Postgres. SQS depth alone does not contain the full backlog.

Worker concurrency is local to one process. For the durable global limit enforced by queue({ concurrencyLimit }), see limit concurrency. Maintenance reserves capacity and requests delivery for bounded runs. Its cadence controls how quickly those runs enter or refill the queue. Use a supervised maintenance service when bounded work needs prompt dispatch.

Understand consumer behavior

The SQS adapter owns receive batches, visibility renewal, FIFO order, and message settlement. Core receives delivery bytes, cancellation, and an optional provider message id.

Worker concurrency limits active deliveries across selected queues. Receives use available capacity, up to SQS's ten-message maximum. Each queue starts with one 20-second long poll. It adds overlapping receives when responses show demand and returns to one after an empty response. Each queue bounds received messages plus outstanding receive reservations by the configured concurrency. Multiple queues may therefore hold waiting messages, with visibility renewal, while sharing the global handler limit. An empty queue's long poll does not reserve handler slots from other queues.

Completed messages are deleted in batches without waiting for the slowest handler from their receive batch. Visibility starts at 300 seconds. Renewal covers messages while they wait, run, and await deletion. One queue-local coordinator groups due renewals into SQS batches of up to ten messages. FIFO messages run in order within each message group; independent groups can progress concurrently. An unresolved delivery, visibility renewal, or delete blocks later messages in its group from that received batch. Shutdown aborts receives and signals handlers, then waits for accepted work and acknowledgements to settle.

FIFO publish order covers messages SQS accepts. A failed batch entry has no broker position. If Runlane retries it later, that retry follows messages SQS already accepted. This matches the transport's same-index per-message outcome contract.

If you use createSqsDeliveryConsumer() directly, maxNumberOfMessages also limits delivery capacity. Standard messages may run at the same time. Set it to 1 for serial work. The Lambda event source still controls Lambda batches.

Estimate cost

Measure the fixed cost of keeping the deployment online and the extra cost of processing work. Report both numbers because a cheap task can still run on an expensive idle deployment.

cost per million successful tasks =
  measured cost during the test
  / successful tasks during the test
  x 1,000,000

measured cost =
  Postgres compute, storage, I/O, and backups
  + Lambda requests and duration, or worker compute
  + Provisioned Mode event-poller EPU-hours, when enabled
  + SQS request units
  + database proxy
  + logs and metrics

For SQS, count every send, receive, delete, and visibility API action. Include repeated actions caused by retries and ReceiveMessage calls that return no records. Batch fill and message size affect billable request units. Use current regional prices for SQS, Lambda, and Fargate.

Give maintenance its own process

For a scheduled Lambda, create src/handlers/runlane-maintenance.ts:

import { runlane } from '../runlane.js'

const runMaintenance = runlane.createMaintenanceHandler()

export async function handler() {
  await runlane.start()

  return runMaintenance()
}

Point an EventBridge schedule or equivalent scheduler at this handler. Do not run maintenance inside each SQS record handler. Maintenance creates scheduled runs, wakes due retries and waits, recovers leases, finishes cancellations, and publishes pending outbox messages.

For an always-on maintenance process, use the supervised service shown in run maintenance.

Check the deployment

Trigger one run from an application process that imports runlane from src/runlane.ts. Confirm its SQS wakeup reaches the selected consumer and ends in a stored terminal result. Replay the message and confirm it does not execute a second job.

Test partial batch failure and dead-letter behavior in an isolated AWS account or supported compatible service.

If SQS has messages but runs do not execute, check the queue binding, environment, task catalog, and delivery failure observer.

If the outbox is pending but SQS is empty, inspect maintenance and publishing. Do not create replacement runs.