Apache Kafka

START HERE: Learn Apache Kafka 2.0 Ecosystem, Core Concepts, Real World Java Producers/Consumers & Big Data Architecture

Last updated 2022-01-10 | 4.7

- Understand Apache Kafka Ecosystem
- Architecture
- Core Concepts and Operations
- Master Concepts such as Topics
- Partitions
- Brokers
- Producers
- Consumers
- Start a personal Kafka development environment

What you'll learn

Understand Apache Kafka Ecosystem
Architecture
Core Concepts and Operations
Master Concepts such as Topics
Partitions
Brokers
Producers
Consumers
Start a personal Kafka development environment
Learn major CLIs: kafka-topics
kafka-console-producer
kafka-console-consumer
kafka-consumer-groups
kafka-configs
Create your Producers and Consumers in Java to interact with Kafka
Program a Real World Twitter Producer & ElasticSearch Consumer
Extended APIs Overview (Kafka Connect
Kafka Streams)
Case Studies and Big Data Architecture
Practice and Understand Log Compaction

* Requirements

* A recent Windows / Mac / Linux machine with minimum 4GB of RAM
* 5 GB of disk space
* Some understanding of Java Programming
* Good to have knowledge about Linux command line
* Desire to learn something awesome and new!

Description

UPDATE SEPTEMBER 2018: Course newly recorded with Kafka 2.0!

Welcome to the Apache Kafka Series! Join a community of 20,000+ students learning Kafka.

Apache Kafka has become the leading distributed data streaming enterprise big data technology. Kafka is used in production by over 33% of the Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart and LinkedIn.

To learn Kafka easily, step-by-step, you have come to the right place! No prior Kafka knowledge is required

===============================

If you look at the documentation, you can see that Apache Kafka is not easy to learn...

Thanks to my several years of experience in Kafka and Big Data, I wanted to make learning Kafka accessible to everyone.

We'll take a step-by-step approach to learn all the fundamentals of Apache Kafka.
At the end of this course, you'll be productive and you'll know the following:

  • The Apache Kafka Ecosystem Architecture

  • The Kafka Core Concepts: Topics, Partitions, Brokers, Replicas, Producers, Consumers, and more!

  • Launch your own Kafka cluster in no time using native Kafka binaries – Windows / MacOS X / Linux 

  • Learn and Practice using the Kafka Command Line Interface (CLI)

  • Code Producer and Consumers using the Java API 

  • Real world project using Twitter as a source of data for a producer and ElasticSearch as a sink for our consumer

  • Overview of Advanced APIs (Kafka Connect, Kafka Streams)

  • Real World Case Studies and Big  Use cases

  • Overview of Advanced Kafka for Administrators

  • Advanced Topic Configurations

  • Annexes (starting a Kafka cluster locally, using Docker, etc...)

Note: The hands-on section is based on Java, which is the native Kafka programming language. But, good news! Your learning in Java will be completely applicable to other programming languages, such as Python, C#, Node.js or Scala, and Big Data frameworks such as Spark, NiFi or Akka

===============================

The Kafka for Beginners course is the best place to start your Kafka learning journey... Ask other students!

  • 5 stars: "That's one of the most high-quality on-line courses I ever took" – Paul L.

  • 5 stars: "This training was awesome and learned many things about KAFKA though having 0 years of experience in this technology" – Puneet G.

  • 5 stars: "Outstanding on all fronts! I wish all courses were that well presented. I especially appreciate the hands-on sections. I was able to get everything up and running with ease. Highly recommend this course and this instructor!" – David G.   

You can take this course risk-free and if you don't like it, you can get a refund anytime in the first 30 days!

===============================

Instructor

My name is Stephane Maarek, and I'll be your instructor in this course. I teach about Apache Kafka, the Kafka ecosystem and Kafka Certifications with my focus always on helping my students improve their professional proficiencies. I am also the co-founder of Conduktor: an enterprise Apache Kafka platform & UI to help everyone use Kafka.

Throughout my career in designing and delivering these certifications and courses, I have already taught 1,000,000+ students and gotten 350,000+ reviews!

With Apache Kafka becoming much more than a buzzword out there, I've decided it's time for students to properly learn how to be a Kafka professional. So, let’s kick start the course! You are in good hands!

===============================

This Course Also Comes With:

  • Lifetime Access to All Future Updates

  • A responsive instructor in the Q&A Section

  • Links to interesting articles, and lots of good code to base your next applications onto

  • Udemy Certificate of Completion Ready for Download

This is the course that could improve your career!

Apache Kafka is a skill in high demand and there are not enough people to fulfill all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies! You can be the next!

I hope to see you inside the course!

=======================

Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:

  • Learn Kafka for Beginners v2 (great to start)

  • Kafka Connect Hands On Learning

  • Kafka Streams for Data Processing

  • KSQL on ksqlDB - Hands On!

  • Kafka Cluster Setup & Administration

  • Confluent Schema Registry & Kafka REST Proxy

  • Kafka Security (SSL SASL ACL)

  • Kafka Monitoring and Operations

Happy learning!

Who this course is for:

  • Developers who want to learn the Apache Kafka Fundamentals, start a cluster and write their first application
  • Architects who want to understand how Apache Kafka fits into their solution architecture
  • Anyone looking to learn the full theory of how Apache Kafka works as a distributed system

Course content

20 sections • 129 lectures

Welcome! Preview 02:20

Apache Kafka in 5 minutes Preview 05:20

Course Objectives Preview 01:47

Important Message Preview 00:24

Kafka Theory Overview Preview 00:29

Topics, Partitions and Offsets Preview 06:47

Brokers and Topics Preview 03:19

Topic Replication Preview 04:12

Producers and Message Keys Preview 05:25

Consumers & Consumer Groups Preview 04:37

Consumer Offsets & Delivery Semantics Preview 04:39

Kafka Broker Discovery Preview 02:23

Zookeeper Preview 02:42

Kafka Guarantees Preview 01:09

Theory Roundup Preview 01:58

Quiz on Theory

Important: Starting Kafka Preview 00:23

FAQ for Setup Problems Preview 00:44

Mac OS X - Download and Setup Kafka in PATH Preview 04:44

Mac OS X - Using brew Preview 01:29

Mac OS X - Start Zookeeper and Kafka Preview 05:38

Mac OS X - Summary Preview 00:19

Linux - Download and Setup Kafka in PATH Preview 05:34

Linux - Start Zookeeper and Kafka Preview 04:56

Linux - Summary Preview 00:21

Windows - Download Kafka and PATH Setup Preview 08:13

Windows - Start Zookeeper & Kafka Preview 04:59

Windows - Summary Preview 00:25

CLI Introduction Preview 00:23

WINDOWS WARNING: PLEASE READ Preview 00:19

Kafka Topics CLI Preview 07:32

Kafka Console Producer CLI Preview 07:08

Kafka Console Consumer CLI Preview 03:21

Kafka Consumers in Group Preview 06:29

Kafka Consumer Groups CLI Preview 04:13

Resetting Offsets Preview 04:07

CLI Options that are good to know Preview 00:13

KafkaCat as a replacement for Kafka CLI Preview 00:09

Quiz on CLI

What about UIs? Conduktor Preview 00:28

Conduktor - Demo Preview 06:03

Intro to Kafka Programming Preview 00:25

Installing Java 11 & IntelliJ Community Edition Preview 00:10

Creating Kafka Project Preview 06:24

Java Producer Preview 10:49

Java Producer Callbacks Preview 06:52

Java Producer with Keys Preview 06:08

Java Consumer Preview 11:49

Java Consumer inside Consumer Group Preview 06:29

Java Consumer Seek and Assign Preview 04:22

Client Bi-Directional Compatibility Preview 01:17

Configuring Producers and Consumers Preview 00:06

Quiz on Java Programming 101

Producer and Advanced Configurations Overview Preview 00:28

Twitter Setup Preview 04:44

Producer Part 1 - Writing Twitter Client Preview 10:13

Producer Part 2 - Writing the Kafka Producer Preview 07:46

Producer Configurations Introduction Preview 00:48

acks & min.insync.replicas Preview 06:15

retries, delivery.timeout.ms & max.in.flight.requests.per.connection Preview 04:45

Idempotent Producer Preview 04:29

Producer Part 3 - Safe Producer Preview 05:15

Producer Compression Preview 04:44

Producer Batching Preview 04:51

Producer Part 4 - High Throughput Producer Preview 03:07

Producer Default Partitions and Key Hashing Preview 02:24

[Advanced] max.block.ms and buffer.memory Preview 02:30

Quiz on Producer Configurations

Refactoring the Project Preview 03:04

Consumer and Advanced Configuration Overview Preview 00:27

Setting up ElasticSearch in the Cloud Preview 03:13

ElasticSearch 101 Preview 05:47

Consumer Part 1 - Setup Project Preview 08:22

Consumer Part 2 - Write the Consumer & Send to ElasticSearch Preview 06:21

Delivery Semantics for Consumers Preview 04:11

Consumer Part 3 - Idempotence Preview 07:45

Consumer Poll Behaviour Preview 02:58

Consumer Offset Commit Strategies Preview 02:44

Consumer Part 4 - Manual Commit of Offsets Preview 04:39

Consumer Part 5 - Performance Improvement using Batching Preview 05:39

Consumer Offsets Reset Behaviour Preview 02:06

Consumer Part 6 - Replaying Data Preview 01:58

Consumer Internal Threads Preview 03:01

Quiz on Consumer Configurations

Kafka Connect Introduction Preview 04:47

Kafka Connect Twitter Hands-On Preview 08:07

Kafka Streams Introduction Preview 03:20

Kafka Streams Hands-On Preview 11:57

Kafka Schema Registry Introduction Preview 04:41

Which Kafka API should I use? Preview 00:09

Quiz on Kafka Extended APIs

Choosing Partition Count & Replication Factor Preview 08:36

Kafka Topics Naming Convention Preview 00:08

Case Study - MovieFlix Preview 05:08

Case Study - GetTaxi Preview 04:18

Case Study - MySocialMedia Preview 05:32

Case Study - MyBank Preview 03:41

Case Study - Big Data Ingestion Preview 01:48

Learn how the big data architecture can be used for ingesting data with Kafka

Case Study - Logging and Metrics Aggregation Preview 01:59

Kafka Cluster Setup High Level Architecture Overview Preview 02:43

Kafka Monitoring & Operations Preview 02:58

Kafka Security Preview 05:01

Kafka Multi Cluster & MirrorMaker Preview 06:48

Changing a Topic Configuration Preview 04:42

Segment and Indexes Preview 05:11

Log Cleanup Policies Preview 02:30

Log Cleanup Delete Preview 02:07

Log Compaction Theory Preview 06:04

Log Compaction Practice Preview 07:02

min.insync.replicas reminder Preview 03:04

Unclean Leader Election Preview 01:17

Annex 1 - Overview Preview 00:33

Starting Kafka with the Confluent CLI Preview 02:59

Starting a multi broker Kafka Cluster using Binaries Preview 06:32

Note on Docker Setup Preview 00:31

Start Kafka Development environment using Docker Preview 04:20

Starting a multi broker Kafka Cluster using Docker Preview 02:55

Kafka Advertised Host Setting Preview 03:25

Starting Kafka on a Remote Machine Preview 05:42

What's Next? Preview 01:23

THANK YOU! Preview 01:32

Bonus Lecture: Student Special Coupons for my Other courses Preview 01:04