Somebody said “we need Kafka”.

Understand it one step at a time

The short runs these in order in about 32 seconds. Here they are written out — pick any step to jump the short straight to it.

1. 1

   Somebody said “we need Kafka”

   It gets said in design reviews constantly, and it is an answer to a question nobody asked out loud.

   design review · claim made · unexamined

2. 2

   Kafka is a log, not a queue

   Service Bus hands you one message and waits. Kafka gives you a position in a file that keeps growing.

   messages · or · offsets

3. 3

   What do you actually need?

   Only two answers matter here: do a piece of work exactly once, or read the same stream many times.

   do work once · or · read a stream

4. 4

   Work needs a broker

   Lock a message, complete it, dead-letter it if it will never work. Kafka makes you build every bit of that.

   PeekLock · DLQ · sessions · scheduled

5. 5

   Kafka caps your parallelism

   One consumer per partition, per group. Eight partitions means eight consumers, and repartitioning is not free.

   8 partitions = 8 consumers, maximum

6. 6

   Streams need a log

   If several teams must read the same events, and read them again next week, you do want a log.

   offsets · retention · replay

7. 7

   On Azure that's Event Hubs

   It exposes a Kafka-protocol endpoint. Point your existing Kafka client at it and change three config lines.

   port 9093 · SASL_SSL · no cluster

8. 8

   So when do you run Kafka?

   When you need Connect, Streams or ksqlDB, exactly-once stream processing, or portability off Azure entirely.

   the ecosystem · not the queue

9. 9

   Broker, log, or cluster

   Three destinations, and only one of them comes with servers you have to keep alive yourself.

   Service Bus · Event Hubs · Kafka

10. 10

    You probably wanted a broker

    And if you genuinely wanted a log, Azure already speaks Kafka without asking you to operate one.

    Service Bus · Event Hubs · Kafka