Transactional Outbox and Idempotent Consumers in Spring Boot: Preventing Double Charges in Event-Driven Payment Systems
How to stop dual-write bugs from double-charging customers, using the transactional outbox pattern, Debezium CDC, and idempotency keys in Spring Boot and Kafka.
A customer wrote in about a doubled hotel booking charge at 2 AM, and the on-call engineer (me, half a bagel into a coffee I'd forgotten to drink) spent the next three hours staring at Kafka consumer lag graphs that looked completely normal. Nothing was lagging. Nothing had crashed. The payment service had published the "charge succeeded" event exactly once per database commit, as far as its own logs were concerned. The problem wasn't in any single service — it was in the gap between two writes that nobody treats as two writes until it bites them.
I've now debugged this exact failure mode at two different companies: once in a fintech building PSD2-compliant payment initiation services, and once at a startup where a naive "save then publish" pattern quietly cost us real money before anyone noticed. Both times the root cause was the same thing with a different name — the dual-write problem. This post is the fix I wish someone had handed me before the first incident, not after it.
The Dual-Write Problem, Stated Plainly
Picture the most common pattern in any Spring Boot service that both persists state and emits an event about that state change:
@Transactional
public void chargeCustomer(ChargeRequest request) {
Payment payment = paymentRepository.save(new Payment(request));
kafkaTemplate.send("payment.charged", new PaymentChargedEvent(payment));
}
This looks fine. It reads fine. It passes code review nine times out of ten, including mine, back when I was three years into my career and hadn't yet been paged awake by it. The problem is that paymentRepository.save writes to Postgres and kafkaTemplate.send writes to Kafka, and those are two separate systems with two separate commit protocols. @Transactional only wraps the database write. There is no two-phase commit spanning a relational database and a Kafka broker in any setup I'd actually recommend running in production.
That gap produces exactly two failure shapes, and I've personally watched both happen in production:
- The database commit succeeds, then the process dies, gets OOM-killed, or loses network before the Kafka send completes. The payment is recorded as charged, but no downstream service — ledger, notification, fraud check — ever hears about it. Money moved, and half the system doesn't know.
- The Kafka send succeeds, but the database transaction then rolls back because of a constraint violation discovered after the send call was already in flight, or the pod gets killed between the send and the commit. Downstream services now believe a charge happened that was never actually recorded. Worse, if a retry layer resends the original HTTP request thinking it failed, you get an event for a charge that exists twice.
Neither of these shows up in your logs as an obvious bug. They show up as a support ticket three weeks later and a very confused finance team doing manual reconciliation in a spreadsheet, which is somehow still how half the fintech industry catches these.
The Outbox Pattern: One Transaction, Not Two
The fix is almost insultingly simple once you see it: stop treating the event as something you send over the network inside the business transaction. Instead, write the event as a row in your own database, in the exact same transaction as the business record. Publishing to Kafka becomes a separate, asynchronous step performed by something that reads that table.
@Entity
@Table(name = "outbox_event")
public class OutboxEvent {
@Id
@GeneratedValue(strategy = GenerationType.UUID)
private UUID id;
@Column(nullable = false)
private String aggregateType; // "Payment"
@Column(nullable = false)
private String aggregateId; // payment.getId().toString()
@Column(nullable = false)
private String eventType; // "PaymentCharged"
@Column(nullable = false, columnDefinition = "jsonb")
private String payload;
@Column(nullable = false)
private Instant createdAt;
@Column
private Instant processedAt; // null until the relay confirms delivery
}
And the service method becomes a single, boring, ACID-guaranteed write:
@Service
@RequiredArgsConstructor
public class PaymentChargeService {
private final PaymentRepository paymentRepository;
private final OutboxEventRepository outboxRepository;
private final ObjectMapper objectMapper;
@Transactional
public Payment chargeCustomer(ChargeRequest request) {
// The idempotency key travels with the request and gets persisted
// right alongside the payment row — this is what makes a retried
// HTTP call a no-op instead of a second charge.
paymentRepository.findByIdempotencyKey(request.idempotencyKey())
.ifPresent(existing -> {
throw new DuplicateChargeAttemptException(existing.getId());
});
Payment payment = paymentRepository.save(Payment.fromRequest(request));
OutboxEvent event = new OutboxEvent();
event.setAggregateType("Payment");
event.setAggregateId(payment.getId().toString());
event.setEventType("PaymentCharged");
event.setPayload(writePayload(payment));
event.setCreatedAt(Instant.now());
outboxRepository.save(event);
return payment;
}
private String writePayload(Payment payment) {
try {
return objectMapper.writeValueAsString(PaymentChargedPayload.from(payment));
} catch (JsonProcessingException e) {
// If we can't serialize the event, we don't want a half-committed
// payment with no corresponding outbox row — fail loud, fail here.
throw new OutboxSerializationException(payment.getId(), e);
}
}
}
Both inserts commit together or neither does. There is no window where a payment exists without a matching outbox row, because Postgres (or whatever relational store you're on — this generalizes to MySQL and most JDBC-backed systems) guarantees atomicity within a single transaction. That's the whole trick. Everything else in this post is about getting the row out of that table and onto Kafka without reintroducing the same problem one layer up.
Getting Events Out: Debezium CDC vs. a Polling Relay
You now have rows sitting in outbox_event that need to become Kafka messages. Two approaches work, and I've shipped both.
Debezium CDC reads the database's write-ahead log directly — for Postgres that's logical replication slots — and streams row changes to Kafka Connect without your application code ever touching the publishing path. I used this at the fintech, and it's the approach I'd default to for anything processing more than a few hundred payments a minute, because it adds essentially zero load to the primary database beyond what WAL replication already does, and latency from insert to Kafka delivery sits in the tens of milliseconds.
The catch is operational weight. You're now running Kafka Connect, managing a Debezium connector configuration, and someone on your team needs to understand logical replication slots well enough to debug why a connector fell behind during a long-running migration that held a lock. If your org doesn't already run Kafka Connect for something else, this is a real cost to take on, not just a YAML file to drop in.
A polling relay is the pragmatic middle ground — a scheduled job in the same Spring Boot app (or a dedicated small service) that queries for unprocessed outbox rows and publishes them itself:
@Component
@RequiredArgsConstructor
public class OutboxRelay {
private final OutboxEventRepository outboxRepository;
private final KafkaTemplate<String, String> kafkaTemplate;
private static final int BATCH_SIZE = 200;
// Fixed delay, not fixed rate — we don't want overlapping runs piling up
// if a Kafka broker hiccup makes one poll take longer than the interval.
@Scheduled(fixedDelay = 500)
@SchedulerLock(name = "outboxRelay", lockAtMostFor = "30s")
public void relayPendingEvents() {
List<OutboxEvent> batch = outboxRepository
.findByProcessedAtIsNullOrderByCreatedAtAsc(PageRequest.of(0, BATCH_SIZE));
for (OutboxEvent event : batch) {
try {
// Keyed by aggregateId so all events for one payment land on
// the same partition — ordering within a payment's lifecycle
// is preserved even with multiple consumer threads downstream.
kafkaTemplate.send("payment.events", event.getAggregateId(), event.getPayload())
.get(5, TimeUnit.SECONDS);
event.setProcessedAt(Instant.now());
outboxRepository.save(event);
} catch (Exception e) {
// Leave processedAt null — the next poll retries it.
// This is exactly how we get at-least-once delivery, and
// exactly why the consumer side needs its own dedup logic.
log.warn("Outbox relay failed for event {}, will retry", event.getId(), e);
}
}
}
}
I lean toward the polling relay for anything under that few-hundred-per-minute threshold, or for teams where I'm the only one who'll end up on-call for the Kafka Connect cluster at 3 AM. It's slower — poll interval plus batch size puts you in the hundreds-of-milliseconds-to-low-seconds range for delivery latency, versus Debezium's near-real-time — but it's one Spring bean, not a second distributed system to operate. I've used ShedLock (the @SchedulerLock annotation above) to keep multiple app instances from double-polling the same rows; without it, horizontal scaling turns your relay into exactly the duplicate-publishing problem you built the outbox to avoid.
Either way, notice the shape of the guarantee you actually get: at-least-once delivery. The relay retries on failure without knowing whether the previous attempt's Kafka write actually landed before the timeout. Debezium's connector offsets have the same at-least-once character on restart. This is not a flaw to engineer away — it's the honest cost of not having a distributed transaction, and it's why the next section exists.
Idempotent Consumers: Dedup Where the Event Lands
At-least-once delivery means your consumer will, eventually, see the same event twice. Maybe the relay retried after a Kafka ack was lost in transit even though the broker committed it. Maybe a rebalance replayed a partition from an earlier offset. It doesn't matter which — what matters is that the consumer, not the producer, is your last line of defense against applying a charge twice.
@Component
@RequiredArgsConstructor
public class PaymentChargedConsumer {
private final ProcessedEventRepository processedEventRepository;
private final LedgerService ledgerService;
@KafkaListener(topics = "payment.events", groupId = "ledger-service")
@Transactional
public void onPaymentCharged(ConsumerRecord<String, String> record) {
UUID eventId = extractEventId(record.value());
// The unique constraint on event_id does the real work here — this
// check-then-insert is a convenience, not the guarantee itself.
if (processedEventRepository.existsById(eventId)) {
log.info("Skipping already-processed event {}", eventId);
return;
}
PaymentChargedPayload payload = parsePayload(record.value());
try {
ledgerService.postCharge(payload);
processedEventRepository.save(new ProcessedEvent(eventId, Instant.now()));
} catch (DataIntegrityViolationException e) {
// Two consumer threads raced past the existsById check on the
// same event — the DB unique constraint is what actually
// prevents the double post, this just turns the crash into a
// clean no-op log line.
log.info("Concurrent duplicate detected for event {}, ignoring", eventId);
}
}
}
The existsById check is a fast path, not the safety mechanism — a unique constraint on event_id in the processed_event table is what actually enforces exactly-once effect, because two threads can both pass the check before either one commits. I learned to stop trusting application-level dedup checks alone after a rebalance-triggered replay beat a race condition I thought I'd already closed with a ConcurrentHashMap-backed in-memory cache — in-memory state doesn't survive a pod restart, and a rebalance can hand the same partition to a different pod instid-second later.
This is also where the idempotency key from the original request earns its keep end to end: the same key that prevented a duplicate HTTP request from creating a second Payment row in the first service now has a parallel concept — the event ID — doing the same job one hop downstream. Idempotency isn't a single check at the API boundary; it's a property you carry through every hop that can replay a message.
Outbox Table Bloat and Relay Thread Tuning
Nobody warns you about this part until the outbox_event table has forty million rows and your nightly VACUUM starts taking long enough to show up in the Postgres slow-query log. A processed outbox row has done its job the moment it's confirmed on Kafka — keeping it around is pure liability.
I run a scheduled cleanup job on a separate cadence from the relay itself:
@Scheduled(cron = "0 0 */6 * * *")
public void purgeProcessedEvents() {
Instant cutoff = Instant.now().minus(Duration.ofHours(24));
int deleted = outboxRepository.deleteByProcessedAtBefore(cutoff);
log.info("Purged {} processed outbox events older than {}", deleted, cutoff);
}
Twenty-four hours of retention gives enough of a window to debug a delivery issue by querying recently processed rows, without the table growing unbounded. For the relay's own thread tuning: a batch size of 200 and a 500ms fixed delay held up fine against roughly 1,200 payments/minute at the startup I mentioned earlier, but when AirAsia-scale traffic patterns came up in a later role — bursts well north of that during fare-sale spikes — a single-threaded relay bean becomes the ceiling on your whole payment pipeline. At that volume, Debezium's log-based streaming stopped being optional. A polling relay has an inherent throughput ceiling set by poll interval × batch size, and no amount of thread-pool tuning changes that math — it just delays hitting it.
Handling In-Flight Duplicates: The 409 Pattern
Stripe's API returns a 409 when an idempotency key is reused while the original request is still processing, and that distinction matters more than it looks. A completed duplicate request should return the original result quietly — the client asked for the same thing twice and gets the same answer. An in-flight duplicate — the client retried before the first attempt even finished — is a different situation, and silently returning "success" for it is how you accidentally tell a client a charge succeeded before you actually know that.
@PostMapping("/charges")
public ResponseEntity<ChargeResponse> charge(@RequestBody ChargeRequest request) {
Optional<Payment> existing = paymentRepository.findByIdempotencyKey(request.idempotencyKey());
if (existing.isPresent()) {
Payment payment = existing.get();
if (payment.getStatus() == PaymentStatus.PROCESSING) {
// Same key, still in flight — tell the client to back off and
// retry rather than letting them believe this call did anything.
return ResponseEntity.status(HttpStatus.CONFLICT)
.body(ChargeResponse.inFlight(payment.getId()));
}
return ResponseEntity.ok(ChargeResponse.from(payment));
}
Payment payment = paymentChargeService.chargeCustomer(request);
return ResponseEntity.ok(ChargeResponse.from(payment));
}
The PROCESSING status row gets written first, synchronously, before the actual charge logic runs — which means the idempotency check in the controller needs to see partial state, not just the fully-settled outcome. I've seen teams skip this and just return the stale "yes it worked" response for an in-flight duplicate, which quietly papers over a double-submit bug on the client side instead of surfacing it where it can get fixed.
Where This Goes in 2026: Agent-Initiated Payments
The newest wrinkle I've been building against this year is agent-initiated payments — an autonomous agent retrying a failed charge call on a customer's behalf, at a frequency and without the human hesitation that used to naturally throttle duplicate attempts. A person who gets an error on a payment form pauses, maybe refreshes, maybe gives up. An agent executing a retry loop can fire three attempts in the time it takes a human to blink, with no built-in sense that it should back off.
That changes the shape of "rare edge case" into "expected steady-state traffic." The in-flight 409 path above, which used to catch an occasional double-click, now needs to handle sustained retry pressure from client code that was never designed with human patience as a rate limiter. I've started treating idempotency-key TTLs and in-flight lock duration as capacity-planning inputs rather than afterthought configuration — if an agent's retry policy backs off at 200ms intervals, your PROCESSING window needs to outlast that, or you'll see 409s stack up into a retry storm that looks like an outage but is actually your own idempotency guard working exactly as designed against traffic nobody sized for.
Closing
I've rebuilt this exact pipeline — outbox table, relay or CDC connector, idempotent consumer, 409 handling — at every payment-adjacent system I've touched since that 2 AM hotel-booking ticket, and I stopped getting paged for duplicate charges once each piece was in place correctly. The pattern isn't exotic; it's one transaction doing honest work instead of two systems pretending to be one. If you're running a dual-write today and haven't been bitten yet, you will be — the only question is whether you fix it before or after the support ticket.
I write about this kind of production payment infrastructure work in more depth over on my experience page, and the systems I've shipped around it are detailed in my projects. If you're dealing with a dual-write bug of your own, reach out — I like these problems more than I probably should.