Kafka is a distributed event streaming platform used for real-time data processing. It’s similar to Celery + RabbitMQ but is more scalable and fault-tolerant.

🚀 Why Use Kafka in Django?

• Handles large-scale real-time data (e.g., stock market prices, chat applications, IoT data).

• Decouples microservices – Producers send data, and multiple consumers can process it independently.

• High throughput & fault-tolerance – It persists messages, making it reliable.

🔗 Kafka vs Celery vs RabbitMQ

Kafka is best when multiple consumers need the same data.

🏗 Step 1: Install Kafka & Zookeeper Locally

Kafka needs Zookeeper to manage brokers.

Windows installation not recommended try with Ubuntu only, as kafka is designed for ubuntu [ if you need window installation steps then comment below I will write separate installation article for it ]

🔧 Install Kafka on Ubuntu

wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz
tar -xzf kafka_2.13-3.5.1.tgz
cd kafka_2.13-3.5.1

🔧 Start Zookeeper & Kafka

# Start Zookeeper (Default Port: 2181)
bin/zookeeper-server-start.sh config/zookeeper.properties &

# Start Kafka Broker (Default Port: 9092)
bin/kafka-server-start.sh config/server.properties &

Why?

• Zookeeper manages Kafka brokers (nodes).

• Kafka listens for producers and consumers on port 9092.

---

📜 Step 2: Create a Django App with Kafka Producer & Consumer

We’ll use Django REST Framework to create a producer that sends messages and a consumer that reads them.

✅ Install Dependencies

pip install kafka-python django djangorestframework

✅ Create a Django App

django-admin startproject kafka_project
cd kafka_project
python manage.py startapp kafka_app

---

🎯 Step 3: Create Kafka Producer (Sends Data)

A producer sends messages to a Kafka topic.

📌 Define Kafka Settings in settings.py

KAFKA_BROKER_URL = "localhost:9092"
KAFKA_TOPIC = "user_activity"

📌 Write Kafka Producer in kafka_producer.py

from kafka import KafkaProducer
import json
from django.conf import settings

class KafkaMessageProducer:
    def __init__(self):
        self.producer = KafkaProducer(
            bootstrap_servers=settings.KAFKA_BROKER_URL,  # Connect to Kafka
            value_serializer=lambda v: json.dumps(v).encode('utf-8')  # Serialize to JSON
        )

    def send_message(self, message):
        self.producer.send(settings.KAFKA_TOPIC, message)  # Send to Topic
        self.producer.flush()  # Ensure the message is sent

# Example usage
producer = KafkaMessageProducer()
producer.send_message({"user": "John", "action": "login"})

📌 Explanation

• KafkaProducer(...) → Creates a producer that connects to Kafka.

• value_serializer=lambda v: json.dumps(v).encode('utf-8') → Converts Python dict to JSON.

• send_message() → Sends a message to the Kafka topic.

• flush() → Forces sending messages immediately.

🎯 Step 4: Create Kafka Consumer (Reads Data)

A consumer listens to messages from a Kafka topic.

📌 Write Kafka Consumer in kafka_consumer.py

from kafka import KafkaConsumer
import json
from django.conf import settings

class KafkaMessageConsumer:
    def __init__(self):
        self.consumer = KafkaConsumer(
            settings.KAFKA_TOPIC,
            bootstrap_servers=settings.KAFKA_BROKER_URL,
            value_deserializer=lambda v: json.loads(v.decode('utf-8')),
            group_id="user_activity_group",  # Consumer Group
            auto_offset_reset="earliest"  # Start from beginning if new
        )

    def consume_messages(self):
        print("Listening for messages...")
        for message in self.consumer:
            print(f"Received: {message.value}")  # Process the message

# Run Consumer
consumer = KafkaMessageConsumer()
consumer.consume_messages()

📌 Explanation

• KafkaConsumer(...) → Connects to Kafka and listens for messages.

• value_deserializer=lambda v: json.loads(v.decode('utf-8')) → Converts JSON to Python dict.

• group_id="user_activity_group" → Ensures multiple consumers don’t read duplicate messages.

• auto_offset_reset="earliest" → Reads messages from the beginning if no offset exists.

• for message in self.consumer: → Infinite loop to process new message

---

🎯 Step 5: Expose API for Kafka Producer

We’ll create a Django REST API that sends Kafka messages.

📌 Create API View in views.py

pythonCopyEditfrom rest_framework.views import APIView
from rest_framework.response import Response
from .kafka_producer import KafkaMessageProducer

class SendMessageView(APIView):
    def post(self, request):
        data = request.data
        producer = KafkaMessageProducer()
        producer.send_message(data)
        return Response({"message": "Sent to Kafka!"})

📌 Define URL in urls.py

pythonCopyEditfrom django.urls import path
from .views import SendMessageView

urlpatterns = [
    path("send-message/", SendMessageView.as_view(), name="send_message"),
]

---

📡 How Kafka Works with Frontend?

Frontend calls the API, which sends data to Kafka. Consumers process the messages and return updates to the frontend via WebSockets.

✅ Frontend Calls API (Sends Data)

fetch("/send-message/", {
    method: "POST",
    body: JSON.stringify({ user: "John", action: "login" }),
    headers: { "Content-Type": "application/json" }
});

✅ Kafka Consumer Processes the Message

Backend listens for Kafka messages and can send updates via WebSockets.

🧐 Where Does Kafka Store Data? How to View It?

Kafka stores messages on disk inside log files. You can view messages using:

# List Kafka Topics
bin/kafka-topics.sh --list --bootstrap-server localhost:9092

# Read Messages from Topic
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user_activity --from-beginning

🎯 Next Steps:

We now have Kafka working. Next, we’ll:

• 🔥 How Kafka Handles Parallel Processing – Topics, Partitions, and Consumer Groups

• Add WebSocket + Django Channels

• Show how Kafka & WebSockets work together

🚀 Stay tuned! I'll write that up next. Let me know if you have any questions so far! 😊

---

Liked it? Please let me know with your comments and likes. ❤️