Moving expensive API work into background jobs

By Milad Akbari ·

A request that generates a large export, calls a slow third-party API or runs a model can take seconds and fail halfway. While it runs, the client waits on an open connection. If a proxy or the client gives up first, the work may still finish on the server, and the client’s retry starts it again.

Moving that work into a background job changes the contract: the API accepts the request, stores it and answers immediately; a separate worker does the work and records the result. The client checks the result later or gets notified.

The examples below are illustrative, not a description of an employer’s systems. They use BullMQ with Redis and were checked locally with Redis 8.10 and BullMQ 5.81. The in-memory map deliberately keeps the example small: the request and worker must run in the same process, and restarting loses the records. A real service needs shared durable storage, authentication and input validation; the queue alone does not provide them.

What stays in the request

The request handler still does everything that can fail fast:

  1. Authenticate and validate the input. A worker should never discover that the request was invalid in the first place.
  2. Store the request in the database with a status such as pending. This record is what the client, the worker and the support team look at.
  3. Put a job on the queue that carries the record’s ID, not a copy of its data. The worker reads the current state when it runs.
  4. Answer with 202 Accepted and a link to the record.

RFC 9110 makes clear that 202 confirms acceptance, not successful completion. A response can explain the current state and point to a status monitor. In this sketch, the returned location represents the URL an HTTP adapter would put in its response; the sketch does not implement an HTTP server or status endpoint.

import { Queue } from "bullmq";

type ExportRecord = {
  id: string;
  status: "pending" | "ready" | "failed";
  fileKey?: string;
};

// Stands in for a database table.
export const exports = new Map<string, ExportRecord>();

export const connection = { host: "127.0.0.1", port: 6379 };

export const exportQueue = new Queue<{ exportId: string }>("exports", {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: "exponential", delay: 1_000 },
    removeOnComplete: 1_000,
    removeOnFail: false,
  },
});

// POST /exports: record the request, queue the work, answer at once.
export async function requestExport(exportId: string) {
  const existing = exports.get(exportId);
  if (existing?.status === "ready") {
    return { status: 200, location: `/exports/${exportId}` };
  }
  // A retry must never reset an existing completed record to pending.
  if (!existing) exports.set(exportId, { id: exportId, status: "pending" });
  // While this job exists, adding the same jobId again is ignored.
  await exportQueue.add("build", { exportId }, { jobId: `export-${exportId}` });
  return { status: 202, location: `/exports/${exportId}` };
}

A custom jobId prevents a duplicate queued job while that ID is retained: BullMQ ignores a job whose ID already exists in the queue. The separate record check preserves a completed result when a request is repeated. Neither check guarantees exactly-once external side effects. Custom IDs must not contain :; the export- prefix also avoids plain integer IDs. Retrying a permanently failed job needs an explicit policy, such as an operator-triggered retry, rather than silently resetting its record.

The gap between the database and the queue

Writing the record and adding the job are two separate operations against two systems, and either can fail on its own:

  • If the record is committed and adding the job fails, the export stays pending forever.
  • If the job is added before the database transaction commits, a fast worker can look for a record that is not visible yet.

There are three common ways to close the gap:

  • A sweeper. A scheduled task re-queues records that have been pending for longer than expected. It is simple and works with any queue, as long as duplicate jobs are harmless.
  • A transactional outbox. The handler writes the record and an outbox row in the same transaction; a relay reads committed outbox rows and adds the jobs.
  • The database as the queue. Workers claim rows with SELECT … FOR UPDATE SKIP LOCKED. PostgreSQL’s documentation notes that skipping locked rows “provides an inconsistent view of the data”, which is unsuitable for general work but fine “with multiple consumers accessing a queue-like table” (SELECT, The Locking Clause).

Whichever you choose, add the job only after the transaction commits. A worker that cannot find the record can then treat it as a permanent error, as the worker below does.

Workers run jobs more than once

Assume any job can run more than once. In BullMQ, a worker that cannot renew its lock, for example because it crashed or blocked the event loop, can have its job marked as stalled and returned for another attempt, or failed once its allowed stall count is exceeded. Retries after errors run the same code again too.

So every job has to be safe to run twice:

  • Check the current state first and skip work that is already done.
  • Write results with keys derived from the input, so a second run overwrites rather than duplicates.
  • Pass an idempotency key to external APIs that accept one, such as Stripe’s API.
  • Separate errors a retry can fix (timeouts, rate limits, a temporarily unavailable service) from errors it cannot (a record that does not exist). BullMQ’s UnrecoverableError moves a job straight to the failed set without further attempts.
import { UnrecoverableError, Worker } from "bullmq";
import { connection, exports } from "./exports";

export function startWorker(buildFile: (exportId: string) => Promise<string>) {
  const worker = new Worker<{ exportId: string }>(
    "exports",
    async (job) => {
      const record = exports.get(job.data.exportId);
      // Retrying cannot fix a missing record.
      if (!record) throw new UnrecoverableError("Export not found");
      // A retry or a second delivery of finished work does nothing.
      if (record.status === "ready") return record.fileKey;

      const fileKey = await buildFile(record.id);
      exports.set(record.id, { ...record, status: "ready", fileKey });
      return fileKey;
    },
    {
      // Required for workers that use ioredis.
      connection: { ...connection, maxRetriesPerRequest: null },
      concurrency: 4,
    },
  );

  // "failed" fires after every failed attempt, not only the last one.
  worker.on("failed", (job, error) => {
    if (!job) return;
    const final =
      error instanceof UnrecoverableError ||
      job.attemptsMade >= (job.opts.attempts ?? 1);
    if (final) {
      exports.set(job.data.exportId, {
        id: job.data.exportId,
        status: "failed",
      });
    }
  });

  return worker;
}

With attempts: 5 and an exponential backoff of one second, BullMQ schedules retry delays of 1, 2, 4 and 8 seconds (Retrying failing jobs); worker availability can make the actual wait longer. Local checks run the snippets with a 20-millisecond backoff and verify five behaviors: a duplicate pending request adds no second job, a repeated completed request preserves ready, two transient failures followed by success leave the export ready after three attempts, a missing record fails after one attempt, and five failures mark the export failed.

Operating the queue

The queue is now part of the system’s state, so it needs the same care as the database:

  • Keep failed jobs. removeOnFail: false keeps them for inspection and replay. Completed jobs can be trimmed; BullMQ otherwise keeps every job forever.
  • Configure Redis for queues. BullMQ’s production guide calls maxmemory-policy noeviction “the only setting that guarantees the correct behavior of the queues” and recommends enabling persistence.
  • Shut down gracefully. Call await worker.close() on SIGTERM, so a deploy lets active jobs finish instead of leaving them to stall.
  • Watch what matters. The number of waiting jobs, the age of the oldest one and the failure rate show trouble before users do. Log the job ID with everything the worker does.
  • Limit concurrency deliberately. Concurrency protects the database and external APIs as much as it speeds up processing. A rate-limited API needs a limiter, not more workers.

Tradeoffs

Background jobs are not free:

  • The client sees the result later, so the interface needs a pending state and a way to learn about completion: polling the status link, a notification or a webhook.
  • There are more moving parts: Redis, worker processes and their deployment, retention and alerting.
  • Jobs processed in parallel finish in any order. Work that must happen in sequence needs one job, a dependency between jobs or a per-key queue.
  • Debugging moves from one request to a chain of events, which makes good logs and the stored status essential.

Work that reliably finishes within a normal response time and has no side effects worth retrying can stay in the request. The move pays off when the work is slow, can fail independently of the request, or competes with requests for the same resources.