COURSE OUTLINE

Day 1 — Introduction, Core Concepts & Architecture

1. Introduction to Event Streaming 

• What is event streaming?
• Why Kafka? Use cases across industries
• Kafka vs Traditional Messaging Systems (RabbitMQ, JMS, Pub/Sub)

2. Kafka Fundamentals 

• Topics
• Partitions
• Brokers
• Replication Factor
• Producers & Consumers
• Consumer Groups

3. Kafka Architecture Deep Dive 

• Cluster architecture
• ZooKeeper vs KRaft (New Metadata Management)
• Message logs & offset management
• Retention policies & compaction
• High availability & fault tolerance

4. Hands-On Labs 

• Installing Kafka (Local Kafka/KRaft setup)
• CLI usage (create topic, produce, consume)
• Working with partitions and offsets

Day 2 — Producers, Consumers & Client Development

1. Kafka Producers 

• Producer API
• Acknowledgements (acks)
• Message Delivery Semantics
• Batching & Compression
• Idempotent producers
• Transactions in Kafka

2. Kafka Consumers

• Consumer API
• Polling Loop
• Offset Commit Strategies
• Rebalancing Concepts
• Consumer Lag & Monitoring

3. Kafka Serialization

• JSON
• Avro
• Protobuf
• Schema Evolution Basics

4. Hands-On Labs

• Build a Java/Python Producer
• Build a Java/Python Consumer
• Integrate with Schema Registry
• Implement idempotent producer

Day 3 – Kafka Connect, Streams, Schema Registry & Integrations

1. Kafka Connect

• Source & Sink Connectors
• Standalone vs Distributed Mode
• Common connectors (JDBC, S3, Elasticsearch, MongoDB)
• Transformations (SMTs)

2. Schema Registry

• Role of schemas in Kafka
• Avro & Protobuf schemas
• Schema evolution rules
• Backward & forward compatibility

3. Kafka Streams

• Streams vs Tables
• Stateless vs Stateful Operations
• Windows, Joins, Aggregations
• Interactive Queries

4. Hands-on Labs

• Setup Kafka Connect & configure connectors
• Build a Kafka Streams application
• Create and evolve schemas in Registry
• Integrate Kafka with a database or API

Day 4 – Operations, Monitoring & Performance Tuning

1. Kafka Administration

• Creating & managing topics
• Partition expansion
• Access control (ACLs)
• Quotas & multi-tenancy
• Data retention & cleanup policies

2. Cluster Management

• Broker configuration
• Multi-broker clusters
• Rack Awareness
• Replication & ISR management

3. Performance Tuning

• Producer & consumer tuning parameters
• Broker performance tuning
• Avoiding hot partitions
• Message sizing best practices

4. Monitoring & Troubleshooting

• Metrics to monitor
• Consumer lag monitoring
• Tools: Prometheus, Grafana, Conduktor, Datadog
• Troubleshooting partitions, offsets, lag, failures

Hands-on Labs 

• Topic admin operations (add partitions, retention policies)
• Monitor Kafka using Prometheus/Grafana
• Stress testing producers and consumers

Day 5 – Enterprise Use Cases, Security, Deployment & Capstone

1. Kafka Security

• SSL/TLS
• SASL/PLAIN, SCRAM
• Authentication & Authorization
• Encryption at rest & in transit
• Best practices for enterprise environments

2. Deploying Kafka at Scale

• On-prem & VM-based clusters
• Kubernetes Deployments (Strimzi, Confluent Operator)
• Cloud-managed Kafka (Confluent Cloud, AWS MSK, Azure Event Hubs)
• Multi-region deployment strategies
• Disaster recovery architectures

3. Advanced Kafka Patterns

• Exactly-once semantics
• Event sourcing with Kafka
• CQRS
• Microservices architectures
• Data pipelines & ETL with Kafka

4. Capstone Project

• Participants build a full real-world end-to-end pipeline:
• Setup topics
• Build producers & consumers
• Integrate Kafka Connect
• Use Schema Registry
• Build Kafka Streams transformation
• Monitor pipeline
• Present findings