Kafka's hidden tax: what 40 migrations tell us about the real cost of streaming
Across 40 enterprise migrations, machines were never the biggest cost. People were. 58% of total streaming spend was engineer time keeping the platform alive.
Every streaming cost conversation starts in the wrong place: the cloud bill. Brokers, storage, cross-zone traffic. Those are real and they are visible, which is exactly why they get all the attention. They are also, in our data, the minority of the cost.
What 40 migrations showed
We looked at forty enterprise migrations from self-managed or vendor-managed Kafka over the last two years and asked each team to reconstruct their fully loaded streaming cost for the twelve months before the move. Not just infrastructure. Everything: on-call hours, upgrade projects, partition rebalancing weekends, the ZooKeeper-to-KRaft migration, the consultant who came in to tune the JVM.
The split was remarkably stable across company size and industry:
- 58% people. Platform engineers, SREs and the fraction of application developers' time spent working around the platform rather than on their product.
- 27% infrastructure. Compute, storage, networking.
- 15% licenses and support.
In other words, the cost most teams try to optimize is a quarter of the total. The cost they rarely measure is more than half.
Where the hours go
When we broke down the people cost, four activities accounted for most of it:
- Keeping it up. Rebalances, broker replacements, disk pressure, consumer lag firefights. The median team had a Kafka-related page every four days.
- Upgrades. Major-version upgrades averaged eleven engineer-weeks, and most teams were two or more versions behind because of it.
- Capacity theater. Over-provisioning "to be safe," then running quarterly exercises to justify or shrink it.
- Schema and client sprawl. Supporting six client libraries across four languages, each with its own quirks.
"We had three engineers whose job description said 'platform.' Their calendar said 'Kafka.'"
Why this is an industry story, not a vendor story
Financial services teams had the highest people share (closer to 65%) because of compliance overhead on every change. Retail and media had the highest infrastructure share, driven by burst traffic. But no sector got the people share below 45%. This is structural: the architecture was designed for a world where engineering time was cheap relative to hardware. That world ended.
What changed after migration
The teams in our sample reported infrastructure savings in the range you'd expect from a more efficient engine. The number that surprised them was people: a 60 to 80 percent drop in platform hours, which most teams redirected to product work rather than headcount reduction. One head of platform described it as "getting a team back without hiring one."
The question to ask
If your streaming budget review only has infrastructure on the slide, you are reviewing a quarter of the cost. Add a line for the hours. Then ask whether the platform you have is designed to minimize them.
People, not machines, are the dominant cost of running streaming infrastructure. Any cost review that stops at the cloud bill is missing more than half the number.
Signals shortlist
- Fully loaded streaming cost split across 40 migrations: 58% people, 27% infrastructure, 15% licenses and support.
- Median team: one Kafka-related page every four days; major upgrades averaged eleven engineer-weeks.
- Financial services showed the highest people share (~65%); no sector fell below 45%.
- Post-migration platform hours fell 60–80%, mostly redirected to product work.
Launch-review draft. Figures in this story are illustrative and drawn from Redpanda field conversations, support data and public sources as noted. They are pending validation by the Redpanda data team before external citation.
- Structured cost interviews with 40 enterprise teams migrating from Apache Kafka or managed Kafka services, 2024–2026.
- Redpanda Customer Success engagement records (anonymized).
- Public total-cost-of-ownership studies on self-managed streaming platforms.
Ground Truth is written and edited by Redpanda and invited contributors. Research methodology and sources are listed in each story.
