Apache Kafka Deep Dive Hands On Using Javabuilt In Scripts

Deep dive in to Apache Kafka concepts and learn to build Kafka producers/consumers using Java.

Last updated 2022-01-10 | 4.4

- Understand all the Kafka concepts and Kafka core internals.
- Learn Kafka internals through practice
- Build Kafka Producer applications using Java

What you'll learn

Understand all the Kafka concepts and Kafka core internals.

Learn Kafka internals through practice

Build Kafka Producer applications using Java

Learn the Kafka Producer Techniques for reliable data delivery

Build Kafka Consumer applications using Java

Advance Kafka Consumer Techniques to handle duplicate processing of records

Consumer Rebalance and different techniques to handle them

Consumer techniques to seek to a specific offset

Learn to build Custom Serializer and Deserializer in Kafka

By the end of the course you will have a complete understanding of Apache Kafka and build Kafka consumer applications using Java

* Requirements

* Basic commands in Unix/Windows Platform.
* Basic Java Knowledge.
* Basics of Windows/Mac Platform

Description

[MAJOR UPDATE : This course was completely redone on March 2020]

Description:

Learn the fundamentals and advanced concepts of Apache Kafka in this course. This course will give you a good understanding of all the concepts through Hands-On

This course will walk you through the Kafka concepts step by step
All the Kafka concepts will be taught through Hands-On
This course will increase your technical expertise in Kafka
This course will give you good exposure to build Kafka Client applications using Java
This course will be an eyeopener for people who think Kafka is a complex system and you will find yourself in a better place after you complete this course
This course will transfer you from no Kafka experience to an experienced Kafka user.

Course Overview:

Getting Started with Kafka

In this section, I will give you all a quick introduction to Apache Kafka, terminologies and different client APIs that are part of Kafka

Download and Install Kafka

In this section, we will download the Kafka distribution from the Kafka Website.

Understanding Kafka Components and its Internals - (Theory + Hands-On)

In this section, we will explore the Kafka internals from a theoretical perspective followed by hands-on exploring the Kafka internals.

Learn about the Topics and Partitions in Kafka
Setup a Local Kafka Cluster with Multiple Brokers
Producer/Consumer messages in the Kafka Cluster
Learn about Consumer Offsets and Consumer Groups
Commit Log and Retention Policy
Learn about Kafka Load Distribution and Fault Tolerance and Robustness

Kafka Producer API - Hands-On

In this section, we will learn about building the Kafka Producer to producer messages into Kafka using the producer API

Setting up the base-project for Producer
Understand how the producer API works
Produce messages to the Kafka topic using the Synchronous/Asynchronous approach
Understand how Kafka producer API works behind the scenes

Kafka Producer API - Guaranteed Message Delivery Configurations

In this lecture, we will learn about the different configurations that guarantee message delivery into Kafka.

Explore some of the producer API Configurations
Configuring the acks and min.insync.replicas for guaranteed message delivery
Override the retries and retry-back-off-ms value\

Kafka Consumer API - Hands On

In this section, we will learn about building Kafka consumers to read messages from Kafka using the Consumer API.

Setting up the base-project for Consumers
Consuming Messages using Consumer API - poll() loop
Build Kafka Consumer using the Consumer API
auto.offset.reset - earliest vs latest

Consumer Groups & Consumer Rebalance

In this section, we will learn about Consumer Groups and Consumer rebalance.

Scaling the Consumer using Consumer Groups
Consumer Rebalance
Consumer Rebalance & max-poll-interval-ms

Consumer Offsets - Default and Manual Offset Management - Hands-On

Committing Offsets - Default Offset Commit Strategy
Manually Committing Offsets
Committing Offsets Synchronously - using commitSync()
Committing Offsets Asynchronously - using commitAsync()
Committing Specific Offsets - using CommitSync()

By the end of this, you will have a complete understanding of how Apache Kafka works and how to build Kafka client applications using Producer and Consumer API.

Who this course is for:

Any one who is willing to learn about Kafka.

Course content

15 sections • 57 lectures

Getting Started with Course 2 lectures

Course Introduction & Objectives Preview 01:48

In this lecture, I will give you all an introduction and objectives for this course.

Pre-Requesities Preview 01:12

In this lecture, I will cover the pre-requisites thats needed to enroll in this course.

Course Slides 1 lectures

Course Slides Preview 00:01

This lecture has the course slides that are used in this course.

Getting Started with Kafka 2 lectures

Introduction to Kafka Preview 06:40

In this lecture, I will explain about How Kafka fits in to the current software architecture and cover some of the examples where Kafka can be used.

Kafka Terminologies and Client APIs Preview 02:45

In this lecture, I will explain about the Kafka Terminologies and different components that are part of Apache Kafka and some of its Client APIs..

Download and Install Kafka 1 lectures

Download and Install Kafka Preview 03:37

In this lecture, we will download the Kafka and explore the different folders inside the Kafka.

Source Code for this Course 1 lectures

Source Code Preview 00:03

This lecture has the source code that's used in this whole course.

Understanding Kafka Components and its Internals - (Theory + Hands On) 12 lectures

Kafka Topics and Partitions - Theory Preview 05:00

In this lecture, we will learn in detail about the Kafka topics, partitions.

Set Up a ZooKeeper/Kafka Broker in Local Preview 05:48

In this lecture, we will set up the Zookeeper and Kafka broker in local.

Create Topic, Produce and Consume Messages using the Command Line Interface(CLI) Preview 05:14

In this lecture, we will learn about creating a topic , produce messages and consume messages from a Kafka Topic.

Produce and Consume Messages with Key Preview 06:39

In this lecture, we will learn about producing and consuming messages with key using the command line interface.

Consumer Offsets Preview 04:02

In this lecture, we will learn about consumer offsets how it plays a major role in keeping a record of resuming processing the records from where it left off in the topic.

Consumer Groups Preview 07:55

In this lecture, we will learn about consumer groups and how it plays a major role in scalable message consumption.

Commit Log and Retention Policy Preview 08:11

In this lecture, we will learn about the partitioned commit log and retention policy of any record thats written in to Kafka.

Kafka as a Distributed Streaming System Preview 04:15

Kafka as a Distributed Streaming SystemIn this lecture, I will explain in detail about how does Kafka behaves as a Streaming System.

Setting up a Kafka Cluster in Local with 3 Kafka Brokers Preview 09:54

In this lecture, we will set up a Kafka Cluster with 3 brokers to simulate a production like environment.

How Kafka Cluster distributes the Client Requests ? - Leader/Follower Preview 05:48

In this lecture, we will learn about how kafka distributes the request between the available brokers.

How Kafka handles Data Loss ? - Replication and In-Sync-Replica(ISR) Preview 09:08

In this lecture, we will learn about replication and In sync Replicas ISR.

Fault Tolerance and Robustness in Kafka Preview 02:39

In this lecture, we will learn about the fault tolerance and robustness in Kafka.

Kafka Producer API - Hands On 8 lectures

Project SetUp Preview 07:09

In this lecture, we will set up the base Kafka Project for the Kafka Producer.

Sending Messages using Producer API Preview 03:44

In this lecture, we will learn about the different options that are available with respect to sending messages into the Kafka Topic.

Build Kafka Producer - Send Messages Synchronously Preview 15:17

In this lecture, we will code the producer using the Kafka Producer.

Add Logging using LogBack Preview 08:26

In this lecture, we will add logging in to our application using LogBack.

Sending Messages Asynchronously Preview 06:49

In this lecture, we will code the Kafka Producer to publish the record in to Kafka asynchronously.

Send Messages with Key Preview 03:41

In this lecture, we will code the producer to send the messages with key.

Programmatic Interface to Produce Messages Preview 06:59

In this lecture, we will build a simple command line interface Java interface to produce messages.

Kafka Producer API - Behind the Scenes Preview 04:16

In this lecture, we will learn about the different layers the producers API goes through before the message is produced to Kafka

Kafka Producer API - Guaranteed Message Delivery Configurations 3 lectures

Kafka Producer API - Guaranteed Message Delivery Configurations Preview 02:46

In this lecture, we will learn about the different configurations that guarantee message delivery in to Kafka.

Configuring acks and min.insync.replicas - Hands On Preview 07:50

In this lecture, we will override the acks value and explore the behavior of the Kafka Producer

Configuring retries and retry.backoff.ms - Hands On Preview 04:06

In this lecture, we will override the retries and retry.backoff.ms value and explore the behavior of the Kafka Producer

Kafka Consumer API - Hands On 4 lectures

Project SetUp Preview 01:53

In this lecture, we will set up the consumers module to build the Kafka consumer application.

Consuming Messages using Consumer API - poll() loop Preview 03:19

In this lecture, we will learn about how to read messages using the consumer API and its behavior behind the scenes.

Build Kafka Consumer using the Consumer API Preview 13:48

In this lecture, we will code the Kafka consumer to read the messages from the Kafka Topic.

auto.offset.reset - earliest vs latest Preview 04:17

In this lecture, I will show a demo on the behavior of the consumer based on the auto.offset.reset value.

Consumer Groups & Consumer Rebalance - Hands On 3 lectures

Scaling the Consumer using Consumer Groups Preview 05:45

In this lecture, we will create another consumer and explore the behavior of consumer groups.

Consumer Rebalance Preview 03:47

In this lecture, we will learn about the consumer rebalance and its importance in Kafka Consumer.

Consumer Rebalance & max.poll.interval.ms Preview 04:16

In this lecture, we will learn about the importance of the max.poll.interval.ms and its impact on consumer rebalance.

Consumer Offsets - Default and Manual Offset Management - Hands On 5 lectures

Committing Offsets - Default Offset Commit Strategy Preview 07:09

In this lecture, we will learn about the consumer's default offset commit strategy and learn about the significance of enable.auto.commit and auto.commit.interval.ms properties.

Manually Committing Offsets Preview 01:31

In this lecture, we will learn about the approaches that are available with respect to committing offsets from our code.

Committing Offsets Synchronously - using commitSync() Preview 04:33

In this lecture, we will code and learn about committing the offsets synchronously using the commitSync() method.

Committing Offsets Asynchronously - using commitAsync() Preview 03:49

In this lecture, we will code and learn about committing the offsets asynchronously.

Committing Specific Offsets - using CommitSync() Preview 05:04

In this lecture, we will code and learn how to commit a specific offset from the consumer

Consumer Rebalance Listeners - Hands On 3 lectures

Introduction to Rebalance Listeners Preview 03:02

In this lecture, I will give you all a quick introduction to Rebalance Listeners and under what scenarios we can use the Rebalance Listeners.

Implement ConsumerRebalanceListener in Kafka Consumer Preview 04:27

In this lecture, we will code and implement the rebalance listener in Kafka Consumer.

Committing Offsets using ConsumerRebalanceListener Preview 04:23

In this lecture we will code and implement the techniques to commit the offsets using the ConsumerRebalance Listener.

Kafka Consumer - seekToBeginning(), seekToEnd() & seek() - Hands On 4 lectures

Introduction to seekToBeginning() & seekToEnd() Preview 04:52

In this lecture, we will learn about the scenarios to use seekToBeginning() and seekToEnd() method which is part of consumer API.

seekToBeginning() & seekToEnd() - Hands On Preview 05:05

In this lecture, we will code and learn about the how to implement the seekToBeginning() and seekToEnd().

Introduction to seek() Preview 03:18

In this lecture, we will explore about the seek() method thats part of the Kafka.

Seek to a specific offset using seek() method in Kafka Consumer - Hands On Preview 11:08

In this lecture we will code and learn about the techniques to seek to a specific offset.

Custom Serializer and Deserializers in Kafka - Hands On 7 lectures

Why Custom Kafka Serializer/Deserializer ? Preview 03:27

In this lecture, we will learn about why do we need custom Kafka Serializer and Deserializer ?

Build a Custom Serializer in Kafka Producer using Jackson Preview 07:49

In this lecture, we will code and learn to build a custom serializer in Kafka Producer

Build Kafka Producer using Custom Serializer Preview 06:21

In this lecture, we will build the Item producer and integrate it with ItemSerializer.

Build a Custom DeSerializer in Kafka Producer using Jackson Preview 04:37

In this lecture, we will code and learn to build a custom deserializer in Kafka Consumer

Build Kafka Consumer using Custom Serializer Preview 04:02

In this lecture, we will build the Item Consumer and integrate it with ItemDeSerializer.

Publish Custom Java Objects using StringSerializer and Jackson ObjectMapper Preview 04:00

In this lecture, we will code and learn about how to use JacksonMapper along with StringSerializer to publish the record using Kafka producer.

Consume Custom Java Objects using StringDeserializer and Jackson ObjectMapper Preview 04:02

In this lecture, we will code and learn about how to use JacksonMapper along with StringDeSerializer to read the record using Kafka Consumer.