Apache Kafka For Developers Using Springboot

Course Slides Preview 00:01

Introduction to Kafka Preview 06:40

In this lecture, I will explain about How Kafka fits in to the current software architecture and cover some of the examples where Kafka can be used.

Kafka Terminologies and Client APIs Preview 02:45

In this lecture, I will explain about the Kafka Terminologies and different components that are part of Apache Kafka and some of its Client APIs.

Download and Install Kafka Preview 03:37

In this lecture, we will download the Kafka and explore the different directories inside the Kafka and its significance.

Kafka Topics and Partitions - Theory Preview 05:00

In this lecture, we will learn in detail about the Kafka topics, partitions.

Set Up a ZooKeeper/Kafka Broker in Local Preview 05:55

In this lecture, we will set up the Zookeeper and Kafka broker in local.

Create Topic, Produce and Consume Messages using the CLI Preview 05:14

In this lecture, we will learn about creating a topic, produce messages and consume messages from a Kafka Topic using Command Line Interface(CLI).

Produce and Consume Messages with Key Preview 06:39

In this lecture, we will learn about producing and consuming messages with key using the command line interface.

Consumer Offsets Preview 04:02

In this lecture, we will learn about consumer offsets how it plays a major role in keeping a record of resuming processing the records from where it left off in the topic.

Consumer Groups Preview 07:55

In this lecture, we will learn about consumer groups and how it plays a major role in scalable message consumption.

Commit Log and Retention Policy Preview 08:11

In this lecture, we will learn about the partitioned commit log and retention policy of any record that's written into Kafka.

Kafka as a Distributed Streaming System Preview 04:15

In this lecture, I will explain in detail about how does Kafka behaves as a Streaming System.

Setting up a Kafka Cluster in Local with 3 Kafka Brokers Preview 09:54

In this lecture, we will set up a Kafka Cluster with 3 brokers to simulate a production like environment.

How Kafka Cluster distributes the Client Requests ? - Leader/Follower Preview 05:48

In this lecture, we will learn about how Kafka distributes the request between the available brokers.

How Kafka handles Data Loss ? - Replication and In-Sync-Replica (ISR) Preview 09:08

In this lecture, we will learn about Replication in Kafka and In-Sync Replicas (ISR).

Fault Tolerance and Robustness in Kafka Preview 02:39

In this lecture, we will learn about the fault tolerance and robustness in Kafka.

Overview of the application - Library Inventory Preview 02:53

In this lecture, I will explain about the application that we are going to build as part of this course.

Source Code Preview 00:03

In this lecture, I will attach the source code for this course.

Setting up the Base project for Library Event Kafka producer Preview 05:17

In this lecture, we will set up the spring boot project for the Library Event Spring Boot Kafka project.

Build the Library Event Domain Preview 03:35

In this lecture, we will build the domain object for the Library Event.

Create the POST endpoint "/libraryevent" Preview 05:50

In this lecture, we will build the post endpoint to publish the Library Event.

Introduction to Spring KafkaTemplate to Produce Messages - Theory Preview 06:57

In this lecture, we will learn about KafkaTemplate which is part of "spring-kafka" to produce Messages.

Configure KafkaTemplate using SpringBoot Profiles - application.yml Preview 06:44

In this lecture, we will code and learn "how to configure" a KafkaTemplate using Spring Boot AutoConfiguration.

How Spring Boot AutoConfiguration Works? - Kafka Producer Preview 04:54

In this lecture, we will code and learn about "How the SpringBoot AutoConfiguration" works ?

AutoCreate TOPIC using KafkaAdmin Preview 06:31

In this lecture, we will auto-create the Kafka topic using the KafkaAdmin configuration.

Build LibraryEvents Producer using KafkaTemplate - Approach1- Part1 Preview 12:32

In this lecture, we will code the LibraryEvents Producer and integrate KafkaTemplate to produce Messages in to the library-events topic asynchronously.

Build LibraryEvents Producer using KafkaTemplate - Approach1- Part2 Preview 04:41

In this lecture, we will code the LibraryEvents Producer and integrate KafkaTemplate to produce Messages into the library-events topic asynchronously.

LibraryEvents Producer API - Behind the Scenes Preview 04:55

In this lecture, I will explain the behavior of the "POST" endpoint "/libraryevent"

Build LibraryEvents Producer using KafkaTemplate - Approach2 Preview 08:11

In this lecture, we will code the LibraryEvents Producer and integrate KafkaTemplate to produce Messages into the library-events topic synchronously.

Build LibraryEvents Producer using KafkaTemplate - Approach3 Preview 07:09

In this lecture, we will code the LibraryEvents Producer to produce Messages into the library-events topic asynchronously using the ProducerRecord.

Sending KafkaRecord With Headers using KafkaTemplate Preview 03:35

In this lecture, we will code the kafkaTemplate to publish the ProducerRecord with KafkaHeaders.

Add LibraryEvent Type - NEW, UPDATE Preview 03:33

In this lecture we will add the Library Event Type to the LibraryEventDomain

Introduction to Automates Tests Preview 02:07

In this lecture , I will give you all a quick introduction to Automated Tests.

Integration Test for POST endpoint - "/v1/libraryevent" - Part 1 Preview 03:31

In this lecture, we will write the integration test for the POST endpoint

Integration Test for POST endpoint - "/v1/libraryevent" - Part 2 Preview 10:33

In this lecture, we will write the integration test for the POST endpoint

Embedded Kafka - Introduction , Configure and Integrate in JUnit Preview 07:56

In this lecture, we will learn about the need for Embedded Kafka and how it helps with the Integration testing.

Integrate Kafka Consumer in JUnit and Consume the record Preview 13:03

In this lecture, we will learn about how to spin up a Kafka Consumer in JUNIT and consume the published record.

Introduction to Unit Testing Preview 01:32

In this lecture , I will give you all a quick introduction to Unit Tests.

Unit Test POST "/libraryevent" endpoint using MockMVC Preview 08:57

In this lecture, we will code and learn how to write unit tests for the post endpoint using MockMVC.

Adding Validations to the LibraryEvent Payload Preview 04:26

In this lecture, we will code and learn about how to add validations to the request body,

Custom ErrorMessages using ControllerAdvice Preview 08:54

In this lecture, we will code and learn about how to add Custom Error Messages to the HttpResponse.

Write JUnit Test for Kafka Producer - OnFailure() Preview 10:13

In this lecture, we will code and learn to write unit test case for a error scenario using JUnit.

Write JUnit Test for Kafka Producer - OnSuccess() Preview 06:55

In this lecture, we will code and learn to write unit test case for a successful scenario using JUnit.

Create the PUT endpoint - "v1/libraryEvent" Preview 08:06

In this lecture, we will build the put endpoint to publish the library events updates in to the Kafka topic.

Assignment - Integration Test PUT "/libraryevent" endpoint using Embedded Kafka

Write an integration test for the PUT endpoint "/v1/libraryevent" using EmbeddedKafka. Use the POST endpoint integration test as an example.

Assignment - Unit Test PUT - "/libraryevent" endpoint using MockMVC

Write the Unit Test for the PUT controller endpoint "/v1/libraryevent".

Kafka Producer - Important Configurations Preview 02:46

In this lecture, we will code and learn how to override the default producer config values in Spring Boot using AutoConfiguration.

Configure the "acks" value as "All" - Hands On Preview 02:58

In this lecture, we will code and learn how to override the default producer config values in Spring Boot.

Set Up the Library Events Consumer Base project Preview 05:15

In this lecture, we will set up the base consumer project library-events consumer.

Introduction to Spring Kafka Consumer - Theory Preview 04:23

In this lecture, we will learn about different Spring Kafka Terminologies which are necessary to configure a Kafka Consumer.

Configure Kafka Consumer using SpringBoot Profiles - application.yml Preview 05:31

In this lecture, we will code and learn about how to configure a Kafka Consumer using Spring Boot AutoConfiguration.

Build Kafka Consumer using @KafkaListener Annotation Preview 07:35

In this lecture, we will code and learn about how to configure a Kafka Consumer using the @KafkListener Annotation.

How Spring Boot AutoConfiguration Works ? - Kafka Consumer Preview 06:41

In this lecture, I will explain about how Spring Boot Auto Configuration works behind the scenes for Kafka Consumer.

Consumer Groups and Rebalance- Hands On Preview 05:57

In this lecture, I will demonstrate how Consumer Groups plays a vital role when it comes to scalable message consumption and consumer rebalance.

Default Consumer Offset Management in Spring Kafka - Hands On Preview 04:39

In this lecture, we will code and learn about the Default consumer offset management by Kafka.

Manual Consumer Offset Management - Hands On Preview 07:24

In this lecture, we will learn about how consumer offset can be manually managed from the application.

Concurrent Consumers - Hands On Preview 02:57

In this lecture, I will demonstrate how to achieve scalable message consumption by increasing the concurrency level to greater than 1.

Configuring the H2 In-Memory DB Preview 04:50

In this lecture, we will configure the in-memory DB in library events consumer.

Create LibraryEvent and Book Entity Preview 06:15

In this lecture, we will code the entity classes to map the Library Event and configure the LibraryEvents Repository.

Build Service Layer to process LibraryEvent - ADD Event Type Preview 11:53

In this lectture, we will code the service layer to process the "ADD" LibraryEvent.

Build Service Layer to process LibraryEvent - MODIFY Event Type Preview 07:28

In this lectture, we will code the service layer to process the "MODIFY" LibraryEvent.

Configure Embedded Kafka for Integration Tests Preview 08:06

In this lecture, we will code and learn to set up the Integration test for the Kafka Consumer.

Write the Integration test for posting a new LibraryEvent Preview 09:53

In this lecture, we will code the first integration test for the Library Events Consumer.

Write the Integration test for posting a "UPDATE" LibraryEvent Preview 07:36

In this lecture, we will code the first integration test for the Library Events Consumer for the MODIFY LibraryEventType.

Integration Test for "MODIFY" Library Event - Not a valid library Event

Write the integration test for the Modify Library Event that's going to result in "IllegalArgumentException("Not a valid library Event")"

Integration Tests for Real Databases using TestContainers Preview 02:21

In this lecture, I will talk about the technique for writing integration tests for real databases.

Custom Error Handler Preview 05:45

In this lecture, we will code and learn about implementing the custom error handler in Kafka Consumer.

Retry in Kafka Consumer Preview 08:26

In this lecture, we will code and learn about introducing the Retry in the Kafka Consumer.

Retry SpecificExceptions using Custom RetryPolicy Preview 08:26

In this lecture, we will code and learn about retrying only some specific exceptions.

Recovery/Retry in Kafka Consumer Preview 03:11

In this lecture, we will code and learn about introducing recovery in the Kafka Consumer flow.

Handling Recovery in Kafka Consumer - Part 1 Preview 05:19

In this lecture, we will implement the recovery logic in Kafka Consumer.

Handling Recovery in Kafka Consumer - Part 2 Preview 10:13

In this lecture, we will implement the recovery logic in Kafka Consumer.

Error Handling in Kafka Producer Preview 02:00

In this lecture, we will discuss about the different error handling options available in Kafka Producer.

Retry in Kafka Producer - Broker Not Available Preview 06:00

In this lecture, we will learn about the options that are available to retry a failed record in Kafka Producer.

Retry in Kafka Producer - min.in.sync.replicas Preview 05:47

In this lecture, we will learn about the recovery options that are available from the Kafka Producer end.

Retain/Recover Failed Records in Kafka Producer Preview 02:24

In this lecture, we will learn about the technique to retain the record that failed because of the Kafka Cluster issue.

Why Kafka Security & How SSL works? Preview 08:11

In this lecture, we will discuss about the need for Kafka Security and How SSL Works?

How Enterprises add/manage SSL Certificate ? Preview 02:18

In this lecture, how enterprises request for SSL certificate from Certificate Authority(CA).

Set Up SSL Kafka Security in Local Preview 16:24

In this lecture, we will set up the SSL security in our local Kafka Cluster

Accessing Secured Kafka Cluster using CLI - Producer/Consumer Messages Preview 08:18

In this lecture, we will learn about the techniques to produce and consume messages for a secured cluster using CLI.

2 Way Authentication - Client & Server Authentication Preview 06:31

In this lecture, we will understand and enable 2 way authentication in Kafka Cluster.

Configuring Library Events Producer to access SSL secured Cluster Preview 08:01

In this lecture, we will learn code and learn about accessing the SSL secured topic from our spring boot application.

Configuring Library Events Consumer to access SSL secured Cluster Preview 03:15

In this lecture, we will learn code and learn about accessing the SSL secured topic from our spring boot application.