Kafka Share Groups in Spring Boot: Replacing RabbitMQ-Style Work Queues in Payment Pipelines

Three years ago I sat in a design review arguing that we needed both Kafka and RabbitMQ in the same payment pipeline. Kafka for the event stream — settlement events, ledger updates, the stuff that needed replay and strict ordering. RabbitMQ for the actual work distribution — retrying a failed card authorization, fanning a single payment instruction out to six workers who'd race to pick it up. Two brokers, two client libraries, two sets of on-call runbooks, one very tired platform team maintaining both. That's the trade a lot of us made because Kafka's consumer-group model has a hard ceiling: parallelism is capped at partition count, and only one consumer in a group ever touches a given partition at a time. If you wanted queue semantics — several workers hammering through a backlog, acknowledging records one at a time, failed ones kicked back without blocking everything behind them — you went and got a queue broker.

KIP-932 changes that math. Share groups, landing as "Queues for Kafka," let multiple consumers read from the same partition concurrently and acknowledge individual records instead of committing an offset for the whole partition. I've spent the last few months running this against a payment retry pipeline in a sandboxed Kafka 4.0 cluster, and I want to walk through where it actually replaces a queue broker, where it doesn't, and the parts of the migration that will bite you if you skip past the fine print.

Why Partition Count Was Always the Real Bottleneck

Classic Kafka consumer groups assign each partition to exactly one consumer instance. That's the whole mechanism — it's what gives you ordering within a partition, and it's also what caps your horizontal scaling. Say you've got a payments topic with 12 partitions carrying authorization retry events. You can run at most 12 consumer instances doing useful work; instance 13 sits idle, a cold spare waiting for a rebalance. Worse, if one retry in partition 7 hits a downstream timeout and your consumer is single-threaded per partition, every other message behind it in that partition queues up behind the retry backoff. One slow card issuer's API can stall an entire partition's worth of unrelated merchants.

I hit this exact wall at a fintech client doing OpenBanking PSD2 payment initiation — a problem I've written about in more detail in my PSD2 API post. Payment retries aren't uniform. Some fail in 200ms with a clean decline. Some hang for the full 30-second gateway timeout before giving up. With 12 partitions and a bad afternoon at one acquiring bank, we had partitions sitting nearly idle while others backed up by thousands of messages, because the hanging retries lived in the same partition as perfectly healthy ones.

The usual answer is to over-partition — bump it to 200 partitions so you have headroom for 200 consumers. That works until it doesn't: more partitions means more open file handles on the brokers, longer rebalance times, and replication overhead that scales with partition count regardless of whether you're using all of them. I've run clusters with partition counts in the thousands purely to buy parallelism headroom nobody used 90% of the time, and the controller logs showing rebalance pauses measured in minutes are not a fun thing to explain in an incident retro.

What a Share Group Actually Does Differently

A share group breaks the one-partition-one-consumer rule. Multiple consumers in the same share group can read from the same partition simultaneously, and acknowledgment happens per record, not per offset-commit-for-the-whole-partition. The broker tracks delivery state per record — essentially a lightweight, broker-managed "in-flight" bitmap — so a record can be delivered, acknowledged, released back for redelivery, or rejected outright, independent of its neighbors in the log.

Practically, that means the hanging authorization retry from the earlier example no longer blocks the ones behind it. It gets delivered to whichever consumer picks it up, that consumer takes its 30 seconds, and meanwhile every other message in that same partition is already being processed by other consumers in the share group. This is the queue semantic I used to need RabbitMQ for — competing consumers racing against a shared backlog, individual message acknowledgment, redelivery on failure — except it's sitting on the same Kafka cluster that already holds my event stream, with the same operational tooling, the same monitoring, the same team that already knows how to read a Kafka broker's GC logs at 2am.

Here's a minimal Spring Boot consumer wired to a share group. Spring for Apache Kafka doesn't have first-class share group support as of the version I tested against (more on that caveat below), so this uses the raw ShareConsumer client wrapped in a Spring-managed lifecycle bean:

@Component
public class PaymentRetryShareConsumer implements SmartLifecycle {

    private static final Logger log = LoggerFactory.getLogger(PaymentRetryShareConsumer.class);
    private final ShareConsumer<String, PaymentRetryEvent> shareConsumer;
    private final PaymentRetryProcessor processor;
    private final ExecutorService workerPool;
    private volatile boolean running = false;

    public PaymentRetryShareConsumer(ShareConsumerFactory factory, PaymentRetryProcessor processor) {
        this.shareConsumer = factory.createShareConsumer("payment-retry-workers");
        this.processor = processor;
        // Virtual threads here because each retry call is a blocking HTTP
        // call to an acquiring bank — the pool cost of a stalled thread
        // is now near zero, unlike the 200-thread-pool days.
        this.workerPool = Executors.newVirtualThreadPerTaskExecutor();
    }

    @Override
    public void start() {
        running = true;
        shareConsumer.subscribe(List.of("payment-authorization-retries"));
        Thread.ofVirtual().start(this::pollLoop);
    }

    private void pollLoop() {
        while (running) {
            ConsumerRecords<String, PaymentRetryEvent> records =
                shareConsumer.poll(Duration.ofMillis(500));

            for (ConsumerRecord<String, PaymentRetryEvent> record : records) {
                workerPool.submit(() -> handleRecord(record));
            }
        }
    }

    private void handleRecord(ConsumerRecord<String, PaymentRetryEvent> record) {
        try {
            processor.retryAuthorization(record.value());
            shareConsumer.acknowledge(record, AcknowledgeType.ACCEPT);
        } catch (PoisonMessageException ex) {
            // Permanently malformed payload — don't let it loop forever.
            log.error("Poison message on retry {}: {}", record.value().paymentId(), ex.getMessage());
            shareConsumer.acknowledge(record, AcknowledgeType.REJECT);
        } catch (GatewayTimeoutException ex) {
            // Transient — let the broker redeliver to any available consumer.
            log.warn("Gateway timeout on retry {}, releasing for redelivery", record.value().paymentId());
            shareConsumer.acknowledge(record, AcknowledgeType.RELEASE);
        }
    }

    @Override
    public void stop() {
        running = false;
        workerPool.shutdown();
        shareConsumer.close();
    }

    @Override
    public boolean isRunning() {
        return running;
    }
}

Three acknowledgment types do the work that used to require a dead letter exchange and a retry policy XML block in RabbitMQ: ACCEPT for success, RELEASE to put the record back for another consumer to try, REJECT to mark it permanently failed and move it out of the main delivery path. The delivery count is tracked by the broker — each release increments it — so you can inspect how many times a record has been retried before deciding it's actually a poison message rather than a transiently unlucky one.

The Decision Framework: When Share Groups Replace a Queue Broker, When They Don't

I'd put the decision in front of an architecture review like this.

Share groups win when you need competing-consumer parallelism without caring about global ordering. Payment retry pools are the textbook case — a failed authorization is independent of every other failed authorization. A notification dispatch queue — sending booking confirmation emails and push notifications — is another. Nobody cares if notification 400 goes out before notification 399. You want raw throughput and fair distribution across workers, and that's exactly what share groups give you without the second broker.

Classic consumer groups still win when you need strict per-key ordering. Settlement jobs are the case I'd never move. If you're netting payment instructions for a merchant across a trading day, event order within that merchant's key matters — a reversal has to apply after the original debit, not before it, and a share group's per-record redelivery explicitly gives up the guarantee that message N+1 won't be processed before message N gets acknowledged. I keep settlement topics on keyed partitions with classic consumer groups, full stop. The same goes for anything building an append-only ledger where replay order is the whole point of using Kafka in the first place — that's event sourcing, not work distribution, and share groups solve a different problem.

The middle ground is wider than people expect. Booking modification notifications at an airline — the use case I worked through at scale handling 130 million requests a day — split cleanly: the booking state change itself goes through a classic consumer group keyed by booking reference, because a cancellation processed before a modification is a real bug. But the fan-out to six downstream notification channels (SMS, email, push, loyalty points update, partner airline sync, audit log) is a share group consuming from a separate "booking-events-to-notify" topic that the ordered consumer publishes into after it's done the ordered part. You get strict ordering where it matters and competing-consumer throughput where it doesn't, without running two brokers.

Idempotency Is Not Optional — It Was Never Optional, It's Just More Obvious Now

RabbitMQ veterans already know this, but it's worth saying plainly for anyone coming from pure Kafka consumer-group backgrounds: RELEASE and broker-side redelivery mean a record can be delivered more than once, including after a consumer has already made partial progress on it. A consumer can crash after calling the payment gateway but before calling acknowledge. The broker will redeliver that record to another consumer, which will call the payment gateway again. If your retry handler isn't idempotent, you've just double-charged a customer.

I handle this with an idempotency key table backed by the same database the payment service already writes to — not Kafka's own exactly-once semantics, which protect the broker's view of state but say nothing about your downstream HTTP call to a bank.

@Service
public class PaymentRetryProcessor {

    private final IdempotencyKeyRepository idempotencyRepo;
    private final AcquiringBankClient bankClient;

    public PaymentRetryProcessor(IdempotencyKeyRepository idempotencyRepo, AcquiringBankClient bankClient) {
        this.idempotencyRepo = idempotencyRepo;
        this.bankClient = bankClient;
    }

    @Transactional
    public void retryAuthorization(PaymentRetryEvent event) {
        String idempotencyKey = event.paymentId() + ":" + event.attemptNumber();

        Optional<IdempotencyRecord> existing = idempotencyRepo.findByKey(idempotencyKey);
        if (existing.isPresent()) {
            // Already processed this exact attempt — a redelivery, not a new retry.
            // Trust the stored outcome rather than hitting the bank twice.
            return;
        }

        IdempotencyRecord inFlight = idempotencyRepo.markInFlight(idempotencyKey);
        try {
            AuthorizationResult result = bankClient.retryAuthorization(event.paymentId(), event.amount());
            idempotencyRepo.markComplete(inFlight.id(), result);
        } catch (GatewayTimeoutException ex) {
            // Mark it reversible so a future redelivery doesn't see a false "already processed."
            idempotencyRepo.markFailed(inFlight.id());
            throw ex;
        }
    }
}

The key detail is event.attemptNumber() rather than just event.paymentId() — the producer increments this on every explicit retry it schedules, so a genuinely new retry attempt and a broker-level redelivery of the same attempt are distinguishable. Collapse those two concepts and you'll either skip retries that should have run, or let redeliveries double-process attempts that already succeeded.

Poison Messages and Delivery Counts

Every queue system eventually needs a way to stop retrying something that will never succeed — a malformed payload, a payment ID that doesn't exist in the ledger anymore, a currency code that isn't supported. Share groups track delivery count per record natively, which means you don't need a separate Redis counter or a custom header to answer "how many times has this been tried."

private void handleRecord(ConsumerRecord<String, PaymentRetryEvent> record) {
    int deliveryCount = record.deliveryCount(); // broker-tracked, survives across consumers

    if (deliveryCount >= MAX_RETRY_ATTEMPTS) {
        deadLetterPublisher.send("payment-retry-dlq", record.value());
        shareConsumer.acknowledge(record, AcknowledgeType.ACCEPT);
        log.error("Payment {} exhausted {} delivery attempts, routed to DLQ",
            record.value().paymentId(), deliveryCount);
        return;
    }

    try {
        processor.retryAuthorization(record.value());
        shareConsumer.acknowledge(record, AcknowledgeType.ACCEPT);
    } catch (GatewayTimeoutException ex) {
        shareConsumer.acknowledge(record, AcknowledgeType.RELEASE);
    }
}

I still route exhausted records to an explicit dead letter topic rather than relying purely on REJECT, because a DLQ topic is something an ops dashboard can alert on and a human can replay once the root cause is fixed — a card network outage that resolves an hour later shouldn't mean every payment that failed during that window is gone for good.

Spring Boot 4 Migration Checklist

If you're planning a move, here's the order I'd actually do it in:

1. Confirm your Kafka cluster is on 4.0 with KRaft — share groups are not available on ZooKeeper-mode clusters, and KRaft migration is its own project with its own risks, not something to bundle into the same change window as a consumer rewrite.
2. Pick exactly one topic for the pilot — I used the lowest-blast-radius queue I had, which was notification fan-out, not anything touching money movement, for the first two weeks in production.
3. Instrument delivery count and acknowledgment type as metrics from day one — you want a dashboard showing the ratio of ACCEPT to RELEASE to REJECT before you trust the system, the same way you'd watch a RabbitMQ queue's redelivery rate.
4. Build the idempotency layer before the share consumer, not after — I made the mistake once of shipping the consumer first and bolting on idempotency under incident pressure a week later. Do it in the other order.
5. Keep classic consumer groups for anything keyed and ordered — don't let "we're moving to share groups" become a mandate to migrate everything; audit each topic for whether order-dependence is real or just inherited from habit.
6. Load test redelivery storms specifically — simulate a downstream dependency going fully unavailable for five minutes and watch what happens to delivery counts and broker memory when every in-flight record gets released simultaneously.

The Honest Status Check on Spring Support

Here's the part I won't dress up: as of the Spring Boot and Spring for Apache Kafka versions I've tested, there's no first-class @ShareKafkaListener annotation or ShareConsumerFactory auto-configuration shipping in the framework. The code above wraps the raw Apache Kafka ShareConsumer client in a hand-rolled SmartLifecycle bean because that's genuinely what you have to do right now. I've seen discussion threads referencing planned support, but I haven't found a released version with confirmed, documented share group integration, and I'm not going to claim otherwise just because it would make this post tidier. If you're building on this today, budget time for writing your own lifecycle management and metrics wiring, and revisit the Spring for Apache Kafka release notes before you assume a declarative annotation exists.

That gap is exactly the kind of production detail that doesn't show up until you've actually run the thing — which is also why I keep a running list of what's shipped versus what's still roadmap on my projects page. If you're wrestling with a similar migration and want a second pair of eyes on the architecture, my contact details are on the contact page.

Related Articles

• OpenBanking PSD2 API Development: Lessons from Production
• Java Virtual Threads in Production: How We Replaced 200-Thread Pools with One Line of Config
• Building Microservices at Scale

Run your own pilot on a single low-risk topic before touching anything that moves money — share groups are a genuine architectural win for competing-consumer workloads, but the redelivery semantics deserve a dashboard and a chaos test before they deserve your settlement pipeline.