Case study · 2026
Transaction Ledger Service
An event-driven ledger that never loses or double-applies a transaction, even when it crashes mid-write.
The problem
A ledger has to apply transactions in order, even when they arrive out of order and the service crashes mid-write. This one accepts writes over REST, returns 202 instantly, and applies balances asynchronously through Kafka. Nothing gets lost, nothing gets applied twice.
Crash the consumer mid-write
Browser modelKafka guarantees at-least-once delivery, so a crash means the message comes back. The ledger guarantees each transaction lands exactly once. Arm a crash and watch the redelivered message get skipped instead of double-applied.
This is a simplified model running in your browser, not the live service. It reproduces the behaviour asserted by redeliveredEvent_isAppliedExactlyOnce, a real Testcontainers test that republishes the same event twice against Postgres and Kafka and asserts the balance never moves.
Waiting for transactions...
Invariant
Balances match the ledger: $0
Sum of balances always equals the sum of writes that reached the ledger, no matter how many times a message is redelivered.
Decisions, and what they cost
Every architecture is a set of trade-offs. These are the ones I made, the alternatives I rejected, and why.
Partition Kafka by account ID
vs. one global ordering keyPer-account ordering is all a ledger needs, and it lets consumers scale out. The cost: no global order, and a hot account can skew a partition.
Manual offset acknowledgement
vs. auto-commitOffsets commit only after the balance is durably applied, so a crash means redelivery, not loss. The consumer is idempotent, and the tests prove exactly-once effects.
Backoff retries (1s/2s/4s), then a dead-letter topic that preserves the partition, with an admin replay endpoint
vs. infinite retry or drop-on-failurePoison messages can't block a partition, and nothing silently disappears. Keeping the partition number on the DLT record means per-account grouping survives the detour, and replay runs under its own consumer group so committed offsets stop it redoing old work. The cost is one more thing to monitor.
ErrorHandlingDeserializer wrapping the JSON deserializer
vs. letting deserialization throwA single malformed payload would otherwise crash-loop the listener forever, since the poison record is redelivered before it can ever be skipped. Wrapping it routes the bad message to the DLT and lets the partition keep moving.
Insufficient funds returns a rejection instead of throwing
vs. throwing on any failed transactionA business rejection is a valid outcome, not an incident. Throwing would burn three retries and about seven seconds of head-of-line blocking on that partition, then pollute the DLT with things that were never broken.
Money as integer minor units, plus @Version optimistic locking on accounts
vs. floats and last-write-winsIntegers kill float rounding outright. Locking matters because a transfer credits the counterparty's row from a different partition's consumer thread, so per-account ordering alone does not prevent a lost update. Optimistic locking turns that race into a retry instead of a silent overwrite.
Three independent idempotency layers
vs. trusting one uniqueness checkA unique idempotency key at submission, a processed-state guard at consumption, and an atomic status-plus-balance commit that makes the guard trustworthy. Any one alone has a gap; together at-least-once delivery becomes effectively-once processing. A duplicate key on a still-pending transaction even re-publishes the event, so a client retry doubles as crash recovery.
Evidence it works
31 tests across 8 classes, including 4 Testcontainers integration tests against real Postgres and real Kafka. Each one pins a specific design claim rather than covering lines: async round trip, submission-order application, exactly-once effect under redelivery, and DLT-to-replay recovery.
Async assertions use Awaitility rather than sleeps, so the suite is neither flaky nor artificially slow.
Four custom Micrometer meters feed Prometheus with histogram buckets, so p99 is computed properly rather than approximated, and consumer lag comes from Kafka's own client metrics instead of a hand-rolled estimate.
A 7-panel Grafana dashboard is provisioned from the repo: throughput by outcome, submit-to-posted latency, consumer lag by partition, rejection rate, and dead-letter rate.
Correlation IDs flow from the HTTP thread through the MDC into Kafka headers and out to the consumer thread, so one async request is traceable across threads in every log line.
Status changes broadcast over SSE only after the transaction commits, so the dashboard can never show a state that later rolls back.
One command brings up Postgres 16, Kafka 3.9.1 in KRaft mode, Prometheus, and Grafana with healthchecks; CI runs backend, frontend, and a gated Docker build.
What I'd do differently
I'd add the Kubernetes deployment earlier. The interesting question a ledger has to answer is what breaks during a rolling restart, and how the consumer group rebalance is handled. That's the difference between running a system and operating it. In progress now.