Add a bulk "replay everything" DLQ endpoint

Nobody has started this yet — be first.

Business impact

The existing POST /v1/dlq/{id}/replay only replays one job at a time. The single most common real-world reason a batch of jobs lands in the DLQ together is a shared root cause -- the downstream had an outage, a deploy introduced a bug, a dependency's certificate expired -- and once that root cause is fixed, an on-call engineer typically wants to clear the entire backlog it created in one action, not click "replay" N times by hand while babysitting each response. Without a bulk endpoint, recovering from an outage that dead-lettered hundreds of jobs means either writing an ad-hoc script against the existing single-job endpoint, or spending real minutes doing it by hand -- both add avoidable time-to-recovery on top of whatever caused the outage in the first place.

Problem

POST /v1/dlq/{id}/replay (see handleReplayDLQ in internal/server/handlers_dlq.go) replays exactly one job, identified by ID. There is no way to replay every currently dead-lettered job in a single request; POST /v1/dlq/replay-all doesn't exist, so nothing routes it (net/http's ServeMux returns 404 or, if the URL happens to match /v1/dlq/{id}'s pattern with id="replay-all", a 405 Method Not Allowed, since that pattern is only registered for GET).

Current behavior

POST /v1/dlq/replay-all returns 405 Method Not Allowed, because net/http's ServeMux matches the path against the existing GET /v1/dlq/{id} pattern, treating "replay-all" as an id -- that route was never meant to handle this URL at all.

Expected behavior

POST /v1/dlq/replay-all lists every currently dead-lettered job (dlqStore.List), replays each one through the retry executor the same way handleReplayDLQ already does for a single job (same policy, same downstream call, same "extend attempt history and keep it in the DLQ on continued failure" behavior for jobs that still fail), removes each job that succeeds from the DLQ (leaving each job that still fails right where it was, with its attempt history extended -- a job that fails replay must never just disappear), and responds 200 OK with a summary of what happened (at minimum, how many jobs were replayed successfully and how many still failed -- the exact JSON shape is your call, but it needs to be possible to tell success from failure per job, not just an aggregate count).

Steps to reproduce

Dead-letter several jobs (e.g. via /v1/charges with X-Simulate: always-fail, or with a low X-Max-Attempts override so an operation that would have succeeded exhausts early). POST /v1/dlq/replay-all. // got: 405 Method Not Allowed // want: 200 OK with a per-job success/failure summary, and any job that succeeded on replay removed from the DLQ

Why this matters

This is the same "architectural extension point" idea POST /v1/dlq/{id}/replay already establishes, generalized to the batch case that's actually the common one during real incident recovery. It also raises a genuine design question worth thinking through explicitly (not just coding around): should jobs replay sequentially or concurrently? A DLQ with many jobs replaying sequentially could take a long time if each replay itself retries several times with backoff; replaying concurrently is faster but means the downstream the DLQ was already struggling against now receives a burst of simultaneous retries right as it's recovering -- arguably the worst possible moment for a thundering herd. There's no single "correct" answer required here, but your implementation should reflect a deliberate choice, not an accident of how you happened to write the loop.

Suggested approach

Add the route in internal/server/server.go's Router(), next to the existing POST /v1/dlq/{id}/replay registration. Add the handler in internal/server/handlers_dlq.go, next to handleReplayDLQ -- most of its per-job logic (unmarshal the payload, run retry.DoWithResult, remove-on-success or extend-and-keep-on-failure) can be factored into a shared helper both handlers call, rather than duplicated. Don't forget s.refreshDLQDepth(ctx) after the batch completes, same as the single-job path already does.

Acceptance criteria

POST /v1/dlq/replay-all exists and returns 200.
A dead-lettered job that would succeed under the server's full retry policy is replayed successfully and removed from the DLQ.
A job that still fails on replay remains in the DLQ (not dropped).
go test ./test/ -run TestTask15 -v passes.

Verification

go test ./test/ -run TestTask15 -v

Hints (0/2)

Try it without hints first — the reading is the exercise.

Working on this ticket

Work on a branch named for the ticket — that's what you'll submit.

Branch off your fork

$git checkout -b feat/dlq-replay-all

Fix it and commit

Meet every acceptance criterion, and add a test that would have caught this.

Push the branch

$git push -u origin feat/dlq-replay-all

Submit it below

Paste your fork URL and the branch name, with a short write-up of the root cause.

Submit your fix

Questions

Stuck on something?

Ask about anything unclear in the ticket — the maintainer and anyone who has solved it can answer. Please don't post full solutions.

No questions yet. If something in this ticket reads ambiguously, you are probably not the only one — ask.

The project

Reliability Layer (Idempotency + Retry + DLQ)

Idempotency, retry-with-jitter, and a dead-letter queue, as three standalone Go libraries plus a runnable HTTP demo -- fifteen ways the payments-style plumbing most real systems get subtly wrong quietly breaks.

Fork on GitHub

Download

Setup

Reliability Layer (Idempotency + Retry + DLQ)

git clone <your-fork>
cd GolangDevTraining/reliable
go run ./cmd/server   # memory backend by default -- no Redis or Docker needed
# in another shell:
curl localhost:8080/healthz

The service runs on . Walk through the README's five core scenarios against (plain success, idempotent replay, conflicting-payload rejection, concurrent duplicates collapsing to one execution, and exhausted retries landing in the DLQ) to see the whole system working end to end.