Browser automation

Use a browser task to gather source material for an AI-assisted research job. Runlane coordinates the attempt, queue, durable source results, and final output. Your application supplies search, model, browser, and network policy.

Research up to five sources

Install Playwright and its Chromium build in the worker image:

npm install @runlane/core playwright zod
npm exec -- playwright install chromium

The Playwright and Runlane calls below are real. The marked source-discovery and summarization calls are pseudocode for your search and model providers:

src/tasks/web-research.ts

import { queue, task } from '@runlane/core'
import { chromium } from 'playwright'
import * as z from 'zod'

const allowedHosts = new Set(['docs.example.com', 'www.example.com'])
const researchUrlSchema = z.url().refine((value) => {
  const url = new URL(value)
  return url.protocol === 'https:' && allowedHosts.has(url.hostname)
}, 'URL must use HTTPS and an allowed host')
const researchUrlsSchema = z.array(researchUrlSchema).max(5)
const modelFindingSchema = z.object({
  keyFacts: z.array(z.string()),
  relevant: z.boolean(),
  summary: z.string(),
})
const researchResultSchema = z.object({
  findings: z.array(modelFindingSchema.omit({ relevant: true }).extend({ source: z.url() })),
  sources: z.array(z.url()),
})

export const browserQueue = queue({ name: 'browser', concurrencyLimit: 2 })

export const webResearch = task({
  id: 'browser.web-research',
  queue: browserQueue,
  schema: z.object({ query: z.string().min(1) }),
  output: researchResultSchema,
  maxAttemptDuration: '2m',
  async run({ query }, context) {
    context.signal.throwIfAborted()

    const sources = await context.step.run('find-sources', { output: researchUrlsSchema }, async () => {
      //////////////////////////////////
      // MAKE YOUR API CALLS HERE
      //////////////////////////////////
      const urls = await findResearchUrls({ query, signal: context.signal })

      return researchUrlsSchema.parse(urls)
    })

    const browser = await chromium.launch()
    let closing: Promise<void> | undefined
    const closeBrowser = () => (closing ??= browser.close())
    const abort = () => void closeBrowser()
    context.signal.addEventListener('abort', abort, { once: true })
    const findings: z.infer<typeof researchResultSchema>['findings'] = []

    try {
      context.signal.throwIfAborted()
      const page = await browser.newPage()
      page.setDefaultTimeout(30_000)

      for (const [index, source] of sources.entries()) {
        const finding = await context.step.run(`research-source-${index}`, { output: modelFindingSchema }, async () => {
          await page.goto(source, { waitUntil: 'domcontentloaded' })
          const pageText = (await page.locator('body').innerText()).slice(0, 8_000)

          //////////////////////////////////
          // MAKE YOUR API CALLS HERE
          //////////////////////////////////
          const result = await summarizeResearchPage({ pageText, query, signal: context.signal, source })

          return modelFindingSchema.parse(result)
        })

        if (finding.relevant) {
          findings.push({ keyFacts: finding.keyFacts, source, summary: finding.summary })
        }
      }

      return { findings, sources }
    } finally {
      context.signal.removeEventListener('abort', abort)
      await closeBrowser()
    }
  },
})

The task validates provider-discovered URLs against an HTTPS host allowlist, visits no more than five pages, bounds extracted text, saves each finding under a stable step key, and always closes the browser.

Replace the sample hosts with destinations permitted by your network policy. If arbitrary destinations are required, enforce egress outside the process and block loopback, private, link-local, and cloud-metadata addresses after every redirect and DNS resolution.

Treat pages as untrusted

Treat page text as data, never as trusted model instructions.
Keep browser credentials out of extracted text and model input.
Isolate cookies and storage between customers.
Bound redirects, downloads, page count, response size, runtime, and memory.
Respect site permissions, robots policies, and applicable terms.

Before production

Use a dedicated queue for expensive browser capacity.
Abort browser work when the task is cancelled.
Test cleanup after every failure point.
Return findings and source URLs, not large page bodies.
Make repeated browser actions safe before enabling retries.

See run workers and limit concurrency.

Browser automation

Research up to five sources

Treat pages as untrusted

Before production

On this page