Module 12: Big Data Analytics for Data ScienceBy Ashok Kumar Yadav4 min read

Apache Kafka: Real-Time Data Streaming for Analytics

Batch jobs tell you what happened yesterday. Kafka moves events as they happen, so dashboards, fraud checks and models can react within seconds.

Core Kafka concepts

A topic is a named stream of events. Each topic is split into partitions, and order is guaranteed inside one partition. Events with the same key go to the same partition. A consumer group shares partitions between its members, so adding consumers scales reading. Offsets mark each consumer's position, and replication across brokers protects against machine failure.

Run Kafka locally

A single-node Docker setup is enough for learning. Use the current image and command from the Apache Kafka documentation.

docker run -d --name kafka -p 9092:9092 apache/kafka:latest
docker exec kafka /opt/kafka/bin/kafka-topics.sh --create \
  --topic orders --partitions 3 --bootstrap-server localhost:9092

Produce events

Using a key keeps all events for one customer in order.

# pip install confluent-kafka
import json, time
from confluent_kafka import Producer

p = Producer({"bootstrap.servers": "localhost:9092"})

def done(err, msg):
    if err: print("failed:", err)

for i in range(5):
    event = {"order_id": i, "customer": f"c{i % 2}", "amount": 100 + i, "ts": time.time()}
    p.produce("orders", key=event["customer"], value=json.dumps(event), callback=done)
p.flush()

Consume and aggregate

Commit offsets after processing so a restart resumes where it stopped.

import json
from collections import defaultdict
from confluent_kafka import Consumer

c = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "revenue-job",
              "auto.offset.reset": "earliest", "enable.auto.commit": False})
c.subscribe(["orders"])
totals = defaultdict(float)
try:
    while True:
        msg = c.poll(1.0)
        if msg is None: break
        if msg.error(): print(msg.error()); continue
        e = json.loads(msg.value())
        totals[e["customer"]] += e["amount"]
        c.commit(msg)
finally:
    c.close()
print(dict(totals))

Streaming analytics next steps

For windowed aggregations and joins, use Spark Structured Streaming, Kafka Streams or Apache Flink instead of hand-written loops. Design for duplicates by making consumers idempotent, and use a schema registry to manage event formats safely over time.

import json, time
event = {'device_id':'M-17','temp_c':71.2,'ts':int(time.time())}
payload = json.dumps(event).encode()
print(payload.decode())

{"device_id": "M-17", "temp_c": 71.2, "ts": ...}

seen=set()
def process(event_id, value):
    if event_id in seen: return 'duplicate skipped'
    seen.add(event_id)
    return f'processed {value}'
print(process('e1',42)); print(process('e1',42))

processed 42
duplicate skipped

Frequently asked questions

Primary references

Use these official sources when exact syntax, defaults or definitions matter beyond the examples here.

• Apache Kafka documentation
• confluent-kafka Python client
• Spark Structured Streaming with Kafka