Kafka Streams

Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8

Last updated 2022-01-10 | 4.6

- Write four Kafka Streams application in Java 8
- Configure Kafka Streams to use Exactly Once Semantics
- Scale Kafka Streams applications

What you'll learn

Write four Kafka Streams application in Java 8

Configure Kafka Streams to use Exactly Once Semantics

Scale Kafka Streams applications

Program with the High Level DSL of Kafka Streams

Build and package your application

Write tests for your Kafka Streams Topology

And so much more!

* Requirements

* Good knowledge of Apache Kafka
* A recent computer (Windows / Linux / Mac OS X) with 4GB of RAM
* Good knowledge of Java 8 Programming Language

Description

The new volume in the Apache Kafka Series!

Learn the Kafka Streams data processing library, for Apache Kafka. Join hundreds of knowledge savvy students into learning one of the most promising data processing library on Apache Kafka.

Kafka Streams is the easiest way to write your applications on top of Kafka:

> Easiest way to transform your data using the High Level DSL
> Exactly Once semantics support out of the box!
> Deploy and Scale your Kafka Streams application without a cluster!
> Perform Aggregations, joins, and any operations you may think of using only a few lines of code!
> Built on top of Kafka, for fault tolerance, scalability and resiliency

Hands On & Theory Based Course

Note: This course is based on Java 8, and will include one example in Scala. Kafka Streams is Java based, and therefore is not suited for any other programming language.

Each section can be either theoretical, or a practice section.

> Through practice, you will be challenged by writing your own Kafka Streams application. The solutions will be thoroughly explained, and you will learn some tips on how to use Kafka Streams the best way.

> Through theory, you will learn about all the available API, the inner workings of the library, as well as some exciting concepts such as Exactly Once Semantics!

This course is the first and only available Kafka Streams course on the web. Get it now to become a Kafka expert!

Section outline:

Kafka Streams - First Look: Let's get Kafka started and run your first Kafka Streams application, WordCount
End-to-End Kafka Streams Application : Write the code for the WordCount, bring in the dependencies, build and package your application, and learn how to scale it. This is a complete end to end example
KStream and KTable Simple Operations: Learn all the stateless operations available for the KStream and KTable API
Practice Exercise - Favourite Colour: Practice your newly acquired skills by writing your own Kafka Streams application, Favourite Colour. It will be challenging! Includes a Scala version of the example
KStream and KTable Advanced Operations: Learn all the stateful operations available for the KStream and KTable API
Exactly Once Semantics - Theory: Learn what EOS (Exactly Once Semantics) is, how Kafka 0.11 enables it, and how to activate them in Kafka Streams
Exactly Once - Practice Exercise - Bank Balance: Practice your newly acquired knowledge by writing your own Kafka Streams Exactly Once application, to compute a running Bank Balance for your customers
Testing your Kafka Streams Application: Learn how to test the WordCount Kafka Streams Topology with Kafka Streams v1.1.0

===============================

Instructor

My name is Stephane Maarek, and I'll be your instructor in this course. I teach about Apache Kafka, the Kafka ecosystem and Kafka Certifications with my focus always on helping my students improve their professional proficiencies. I am also the co-founder of Conduktor: an enterprise Apache Kafka platform & UI to help everyone use Kafka.

Throughout my career in designing and delivering these certifications and courses, I have already taught 1,000,000+ students and gotten 350,000+ reviews!

With Apache Kafka becoming much more than a buzzword out there, I've decided it's time for students to properly learn how to be a Kafka professional. So, let’s kick start the course! You are in good hands!

===============================

This Course Also Comes With:

Lifetime Access to All Future Updates
A responsive instructor in the Q&A Section
Links to interesting articles, and lots of good code to base your next applications onto
Udemy Certificate of Completion Ready for Download

This is the course that could improve your career!

Apache Kafka is a skill in high demand and there are not enough people to fulfill all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies! You can be the next!

I hope to see you inside the course!

=======================

Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:

Learn Kafka for Beginners v2 (great to start)
Kafka Connect Hands On Learning
Kafka Streams for Data Processing
KSQL on ksqlDB - Hands On!
Kafka Cluster Setup & Administration
Confluent Schema Registry & Kafka REST Proxy
Kafka Security (SSL SASL ACL)
Kafka Monitoring and Operations

Happy learning!

Who this course is for:

Developers and devops who would like to learn how to write, package, deploy and run a Kafka Streams applications
Architects who would like to understand how Kafka Streams work and its position in the Kafka-centered data pipeline and enterprise architecture
People who fully understand the basics of Kafka
Java 8 and Scala developers only

Course content

11 sections • 76 lectures

Kafka Streams - First Look 5 lectures

What is Kafka Streams? Preview 04:46

Learn what is Kafka Streams at a high level

Course Objective / Prerequisites / Target Students Preview 04:11

Run through some pre-requisites for this course, understand who the target student is

About your Instructor: Stephane Maarek Preview 02:20

Short bio of your instructor, Stephane Maarek!

Running your first Kafka Streams Application: WordCount Preview 12:52

Full end to end run of your first Kafka Streams application.

We will download Kafka, start our own cluster, and run producers, consumers, and our first Kafka Streams application

Kafka Streams vs other stream processing libraries (Spark Streaming, NiFI, Flink Preview 03:00

Short lecture comparing Kafka Streams to other streaming libraries

Code Download 1 lectures

Code Download Preview 00:14

End to End Kafka Streams Application - Word Count 15 lectures

Section Objective Preview 01:45

Description of the Word Count Kafka Streams application

Kafka Streams Core Concepts Preview 03:28

Learn the Kafka Streams Core Concepts, Topologies, Sink, Sources, Processors, Streams

Environment and IDE Setup: Java 8, Maven, IntelliJ IDEA Preview 04:03

Setup Java 8, Maven, IntelliJ IDEA Community Editions so we can get started coding our first Kafka Streams application

Starter Project Setup Preview 07:56

Setup a project starter so we can include the right kafka streams dependencies and get coding!

Kafka Streams Application Properties Preview 07:25

Write our first Kafka Streams code. We will setup the Kafka Streams properties

Java 8 Lambda Functions - quick overview Preview 02:55

Learn about what is a Java 8 Lambda and how we will use them

Word Count Application Topology Preview 14:07

Write our application topology using the newly acquired Java 8 Lambdas.

This is the first time we're dealing with the Kafka Streams API

Printing the Kafka Streams Topology Preview 01:33

Learning how to print the Kafka Streams topology

Kafka Streams Graceful Shutdown Preview 02:03

Add some code so that our Kafka Streams application shuts down gracefully when we stop it

Running Application from IntelliJ IDEA Preview 09:22

Learn how to run our Kafka Streams application straight from IntelliJ

Debugging Application from IntelliJ IDEA Preview 04:06

Learn how to run the IntelliJ debugger to do step through in our code

Internal Topics for our Kafka Streams Application Preview 03:47

Learn about Kafka Streams internal topics (repartition and changelog topics)

Packaging the application as Fat Jar & Running the Fat Jar Preview 04:33

Learn how to use Maven Assembly to package our Kafka Streams application as a fat jar

Scaling our Application Preview 03:59

Learn how to scale a Kafka Streams application

Section Wrap-Up Preview 01:40

Wrap up on all the awesome learning from this section

KStreams and KTables Simple Operations (Stateless) 16 lectures

Section Objectives Preview 01:14

Learn the objective of this section, learning the stateless operations in Kafka Streams

KStream & KTables Preview 03:39

Learn about KStream and KTables

Stateless vs Stateful Operations Preview 01:41

Learn what Stateless and Stateful mean

MapValues / Map Preview 01:35

Learn about the Map and MapValues Kafka Streams Operation

Filter / FilterNot Preview 01:22

Learn about the Filter and FilterNot Kafka Streams Operation

FlatMapValues / FlatMap Preview 02:09

Learn about the FlatMap and FlatMapValues Kafka Streams Operation

Branch Preview 02:45

Learn about the Branch Kafka Streams Operation

SelectKey Preview 01:14

Learn about the SelectKey Kafka Streams Operation

Reading from Kafka Preview 02:16

Learn how to read a KStream or KTable from Kafka

Writing to Kafka Preview 01:52

Learn how to write your KStream or KTable back to Kafka

Streams Marked for Re-Partition Preview 02:52

Learn what operations mark streams for re-partition

Refresher on Log Compaction Preview 17:44

Learn the fundamentals behind Log Compaction. This is a refresher!

KStream and KTables Duality Preview 02:20

Learn what is the KStream and KTable Duality

Transforming a KTable to a KStream Preview 00:41

Learn how to transform a KTable into a KStream in Kafka Streams

Transforming a KStream to a KTable Preview 01:24

Learn how to transform a KStream into a KTable

Section Summary Preview 00:47

Summary of all the learning on Stateless Operations

Practice Exercise - FavouriteColour 5 lectures

FavouriteColour - Practice Exercise Description & Guidance Preview 03:20

Learn about the practice Kafka Streams exercise, favourite colour!

Stuck? Here are some Hints! Preview 02:39

Hints and guidance to conduct the programming of this Kafka Streams application by yourself

Java Solution Preview 08:33

Java solution for the Kafka Streams application FavouriteColour

Running the application Preview 05:46

Running the Favourite Colour Kafka Streams application

Scala Solution Preview 06:20

Learn how to write the same application using Scala

KStreams and KTables Advanced Operations (Stateful) 9 lectures

Section Objective Preview 01:00

Section objective on KStream and KTable stateful operations

KTable groupBy Preview 01:46

Learn about the groupBy operation

KGroupedStream / KGroupedTable Count Preview 02:05

Learn the count operation

KGroupedStream / KGroupedTable Aggregate Preview 04:12

Learn the aggregate operation

KGroupedStream / KGroupedTable Reduce Preview 01:40

Learn the reduce operation

KStream peek Preview 02:08

Learn the peek operation on KStream

KStream Transform / TransformValues Preview 01:12

Learn the Transform / Transform Values operation on KStream

What if I want to write to an external System? Preview 01:40

Learn how Kafka Connect can help you write your transformed data into a sink store / external system

Summary Diagram Preview 01:03

Summary diagram from the Confluent documentation for operations

Exactly Once Semantics - Theory 4 lectures

What's Exactly Once? Preview 06:07

Learn what Exactly Once Semantics are

Exactly Once in Kafka 0.11 Preview 02:41

Learn how Kafka 0.11 solves the problem to have exactly once semantics

What's the problem with at least once anyway? Preview 01:29

Understand the use cases of At Least Once vs Exactly Once

How to do exactly once in Kafka Streams Preview 02:04

Learn how to easily achieve exactly once semantics in Kafka Streams!

Exactly Once Semantics - Practice Exercise - BankBalance 7 lectures

BankBalance - Exercise Overview Preview 02:14

Description for the BankBalance practice Kafka Streams exercise

Kafka Producer Guidance Preview 01:41

Guidance for you to write your Kafka Producer in order to produce random bank transactions

Kafka Producer Solution Preview 14:17

Solution for the Kafka Producer that generates JSON random bank transactions

Kafka Streams Guidance & Hints Preview 01:54

Hints on how to write your Kafka Streams application

Kafka Streams Solution Preview 09:12

Solution to build the Bank Balance Kafka Streams application

Running the BankBalance Application Preview 04:42

Let's run the Bank Balance producer and Kafka Streams application!

Section Summary Preview 00:56

Summary of our learnings to date

Joins - KStream to GlobalKTable example 6 lectures

What are joins in Kafka Streams? Preview 02:49

Learn about Joins in Kafka Streams. Learn when it is or isn't possible to join data

Join Constraints and GlobalKTables Preview 02:53

Understand what is co-partition of data, join constraints, and how global KTables solve the problem