Transaction Ledger Service

A ledger has to apply transactions in order, even when they arrive out of order and the service crashes mid-write. This one accepts writes over REST, returns 202 instantly, and applies balances asynchronously through Kafka. Nothing gets lost, nothing gets applied twice.

1. Partition Kafka by account ID

   Instead of one global ordering key

   Per-account ordering is all a ledger needs, and it lets consumers scale out. The cost: no global order, and a hot account can skew a partition.

2. Manual offset acknowledgement

   Instead of auto-commit

   Offsets commit only after the balance is durably applied, so a crash means redelivery, not loss. The consumer is idempotent, and the tests prove exactly-once effects.

3. Backoff retries (1s/2s/4s), then a dead-letter topic that preserves the partition, with an admin replay endpoint

   Instead of infinite retry or drop-on-failure

   Poison messages can't block a partition, and nothing silently disappears. Keeping the partition number on the DLT record means per-account grouping survives the detour, and replay runs under its own consumer group so committed offsets stop it redoing old work. The cost is one more thing to monitor.

4. ErrorHandlingDeserializer wrapping the JSON deserializer

   Instead of letting deserialization throw

   A single malformed payload would otherwise crash-loop the listener forever, since the poison record is redelivered before it can ever be skipped. Wrapping it routes the bad message to the DLT and lets the partition keep moving.

5. Insufficient funds returns a rejection instead of throwing

   Instead of throwing on any failed transaction

   A business rejection is a valid outcome, not an incident. Throwing would burn three retries and about seven seconds of head-of-line blocking on that partition, then pollute the DLT with things that were never broken.

6. Money as integer minor units, plus @Version optimistic locking on accounts

   Instead of floats and last-write-wins

   Integers kill float rounding outright. Locking matters because a transfer credits the counterparty's row from a different partition's consumer thread, so per-account ordering alone does not prevent a lost update. Optimistic locking turns that race into a retry instead of a silent overwrite.

7. Three independent idempotency layers

   Instead of trusting one uniqueness check

   A unique idempotency key at submission, a processed-state guard at consumption, and an atomic status-plus-balance commit that makes the guard trustworthy. Any one alone has a gap; together at-least-once delivery becomes effectively-once processing. A duplicate key on a still-pending transaction even re-publishes the event, so a client retry doubles as crash recovery.

• 31 tests across 8 classes, including 4 Testcontainers integration tests against real Postgres and real Kafka. Each one pins a specific design claim rather than covering lines: async round trip, submission-order application, exactly-once effect under redelivery, and DLT-to-replay recovery.

• Async assertions use Awaitility rather than sleeps, so the suite is neither flaky nor artificially slow.

• Four custom Micrometer meters feed Prometheus with histogram buckets, so p99 is computed properly rather than approximated, and consumer lag comes from Kafka's own client metrics instead of a hand-rolled estimate.

• A 7-panel Grafana dashboard is provisioned from the repo: throughput by outcome, submit-to-posted latency, consumer lag by partition, rejection rate, and dead-letter rate.

• Correlation IDs flow from the HTTP thread through the MDC into Kafka headers and out to the consumer thread, so one async request is traceable across threads in every log line.

• Status changes broadcast over SSE only after the transaction commits, so the dashboard can never show a state that later rolls back.

• One command brings up Postgres 16, Kafka 3.9.1 in KRaft mode, Prometheus, and Grafana with healthchecks; CI runs backend, frontend, and a gated Docker build.

I'd add the Kubernetes deployment earlier. The interesting question a ledger has to answer is what breaks during a rolling restart, and how the consumer group rebalance is handled. That's the difference between running a system and operating it. In progress now.