Español

Use a Postgres Table Before You Reach for a Queue

You do not need a message queue. You need a table.

The moment someone says "background jobs" or "async work," the reflex is to spin up SQS, RabbitMQ, or Kafka. Now you operate a second stateful system: its own availability, its own failure modes, its own dashboards, its own way to lose messages. For a lot of low-to-medium-volume work, you already have everything you need in the database you're running anyway.

The reflex costs more than it looks

A dedicated broker is real operational surface. You have to provision it, monitor it, secure it, and keep it available. You also introduce a dual-write problem: your app now writes to Postgres and the queue, and keeping those two in sync across failures (did the row commit but the enqueue fail?) is exactly the kind of distributed bug that eats weeks. For a small team doing modest volume, that's a lot of cost for capacity you're not using.

A table is a fine queue

If you already run Postgres, and your producers and consumers live in the same deployment, a plain table plus one clause gets you a correct, durable queue. The key is FOR UPDATE SKIP LOCKED, which lets many workers pull distinct rows concurrently without blocking each other:

CREATE TABLE jobs (
  id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  payload     jsonb NOT NULL,
  status      text  NOT NULL DEFAULT 'pending',
  run_after   timestamptz NOT NULL DEFAULT now(),
  attempts    int   NOT NULL DEFAULT 0,
  created_at  timestamptz NOT NULL DEFAULT now()
);

-- A worker claims one job atomically:
SELECT id, payload
FROM jobs
WHERE status = 'pending' AND run_after <= now()
ORDER BY id
FOR UPDATE SKIP LOCKED
LIMIT 1;

Because the claim and the work-enqueue can happen in the same transaction as your business writes, the dual-write problem disappears: either the row and the job both commit, or neither does. You get durability, retries (bump attempts, set run_after for backoff), and a dead-letter state (status = 'failed') for free, all with tools you already operate and can query with plain SQL.

When to actually graduate

The point isn't "never use a broker" — it's "don't start with one." Reach for SQS/RabbitMQ/Kafka when you hit real limits:

Mature libraries lean on exactly this pattern — Postgres-backed queues are a well-trodden path, not a hack. Graduate when the evidence forces it, and the migration will be obvious.

Power moves

• Use SKIP LOCKED, not naive SELECT ... LIMIT 1 — without it, concurrent workers contend on the same row and serialize.
• Add a partial index on (status, run_after) WHERE status = 'pending' so claiming stays fast as the table grows.
• Consider LISTEN/NOTIFY to wake workers instantly instead of tight polling loops.
• Archive or delete completed rows on a schedule so the hot table stays small.
• Reach for a proven library (e.g. pgmq, graphile-worker, River, Oban, solid_queue) before hand-rolling edge cases like visibility timeouts.

Resources

• FOR UPDATE ... SKIP LOCKED — PostgreSQL docs
• "Choose Boring Technology" — Dan McKinley
• LISTEN / NOTIFY — PostgreSQL docs
• pgmq — a lightweight message queue extension for Postgres