Español

Test Background Jobs for Idempotency and Retries

Your background job will run twice. If you did not test for that, someone gets double charged. That's not pessimism — it's the delivery contract. Sidekiq's own docs say it plainly: "Sidekiq will execute your job at least once, not exactly once." Amazon SQS standard queues make the same promise: at-least-once delivery, and "more than one copy of a message might be delivered." Duplicates aren't a bug in your queue. They're the spec.

Why every queue re-runs your job

Job systems can only guarantee a job wasn't lost by re-sending it whenever they're unsure it finished. The unsure cases are everywhere:

• A worker crashes after doing the work but before acknowledging the message — the queue re-delivers.
• A job outlives its visibility timeout or ack deadline — the queue assumes it died and hands it to another worker.
• A transient failure (network blip, deploy restart, rate limit) triggers the framework's automatic retry.

Run twice means charge twice, email twice, ship twice — unless the second run is harmless. That property has a name: idempotency. Celery's task guide defines the goal exactly: "the function won't cause unintended effects even if called multiple times with the same arguments."

The two tests every job needs

In code, the idempotency test is almost boring — which is the point:

def test_charge_job_is_idempotent(db, payment_gateway):
    charge_order(order_id=42)
    charge_order(order_id=42)          # duplicate delivery

    assert payment_gateway.charge_count(order_id=42) == 1
    assert db.order(42).status == "paid"

The retry test injects the failure the queue will eventually inject for you:

def test_charge_job_recovers_after_crash(db, payment_gateway):
    with payment_gateway.fail_after_charge():   # crash between work and ack
        with pytest.raises(GatewayTimeout):
            charge_order(order_id=42)

    charge_order(order_id=42)                   # the retry
    assert payment_gateway.charge_count(order_id=42) == 1

If either test can't pass, the fix is usually one of three patterns: a unique key on the side effect (e.g. an idempotency key sent to the payment provider), a state check before acting ("skip if already paid"), or making the whole job transactional so a crash rolls back to a clean re-runnable state — Sidekiq's best-practices page calls this pair out directly: make jobs idempotent and transactional.

Power moves

• Test the retry policy, not just the retry. Frameworks expose real knobs — Active Job's retry_on defaults to wait: 3.seconds, attempts: 5 with jitter, and discard_on drops a job instead of retrying it. Assert that the exceptions you expect map to the policy you chose, so a refactor can't silently turn "retry 5 times" into "retry forever."
• In Celery, pair idempotency with acks_late. By default the message is acknowledged before the task runs; acks_late=True acknowledges after it returns, so a crashed worker's job is re-delivered — the docs recommend it specifically for idempotent tasks. Use autoretry_for with retry_backoff for transient errors.
• Don't trust "exactly-once" checkboxes blindly. Google Cloud Pub/Sub offers exactly-once delivery only as an opt-in subscription feature, and redelivery still happens on nacks and expired ack deadlines. Idempotent handlers are the layer that works everywhere.
• Make duplicate runs observable. Log "already processed, skipping" as an info event with the dedup key. When the queue re-delivers in production — and it will — you get a metric, not a mystery.

Now a duplicate delivery is safe behavior, not a support ticket.

Resources

• Sidekiq Best Practices — make jobs idempotent and transactional
• Celery Tasks guide — idempotence, acks_late, retries
• Amazon SQS standard queues — at-least-once delivery
• Active Job retry_on / discard_on API
• Google Cloud Pub/Sub — exactly-once delivery is opt-in