Confluent Schema Registry

Kafka - Master Avro, the Confluent Schema Registry and Kafka REST Proxy. Build Avro Producers/Consumers, Evolve Schemas

Last updated 2022-01-10 | 4.6

- Write simple and complex Avro Schemas
- Create
- Write and Read Avro objects in Java
- Write a Java Producer and Consumer leveraging Avro data and the Schema Registry

What you'll learn

Write simple and complex Avro Schemas

Create

Write and Read Avro objects in Java

Write a Java Producer and Consumer leveraging Avro data and the Schema Registry

Learn about Schema Evolution

Perform Schema evolution using the command line and in Java

Utilize the REST Proxy using a REST Client

* Requirements

* Good knowledge of Apache Kafka
* Some Java knowledge
* A recent computer (Windows / Linux / Mac OS X) with 4GB of RAM
* Understand JSON data format
* Knowledge of REST APIs is better

Description

The new volume in the Apache Kafka Series!

Learn Apache Avro, the Confluent Schema Registry for Apache Kafka and the Confluent REST Proxy for Apache Kafka. Join hundreds of knowledge savvy students into learning some of the most important components in a typical Apache Kafka stack.

Take a look at these student reviews...

5 stars - "Excellent course. A must for Apache Kafka users." – Mayank

5 stars -"Beautiful explanation of Avro and how to use it with Apache Kafka. This knowledge enhancing like all your other courses. We have Apache Kafka from Confluent in our enterprise and this is what I was exactly looking for. Thanks a lot." – Vinod

5 stars - "Stephane Maarek is truly remarkable instructor! His instructional videos do not disappoint, they keep me wanting more and provide exactly the Kafka skills needed to be successful." – Troy

----------------------------

Apache Avro is one of the most powerful and most popular fast data serialisation mechanism with Apache Kafka

> Schema is a first class citizen of the data
> Data serialisation format that is lightweight
> Documentation is embedded in the schema
> Easy to read and write using many languages
> Fast & Big data de-facto data serialisation format

Confluent Schema Registry for Apache Kafka is the de-facto standard way of storing Avro Schemas for your Apache Kafka Topics.

> Stores a versioned history of all your schemas in Apache Kafka
> Supports and enforces conditions for schema evolution (backward, forward, full compatibility)
> Kafka Avro Serialiser and Deserialiser automatically integrate with Apache Kafka and Confluent Schema Registry
> Make your messages much smaller and lighter! Perfect for increasing Apache Kafka throughput and performance

Confluent REST Proxy is the perfect way to communicate for sending Avro data using non Java languages to Apache Kafka.

> Write and Read binary, JSON and Avro data to Apache Kafka using an HTTP REST API
> Interact with Apache Kafka using any programming language (not just Java)
> Consult topic list and topic metadata in Apache Kafka

Hands On & Theory Based Course

Note: This course is comprehensive, and the Apache Kafka related bits do not start before the Kafka Schema Registry section. This is to ensure Avro Schema and Avro in Java is fully understood before moving to the Confluent Schema Registry for Apache Kafka.

This course is the first and only available Kafka Schema Registry and Kafka REST Proxy course on the web. Get it now to become an Apache Kafka expert!

===============================

Instructor

My name is Stephane Maarek, and I'll be your instructor in this course. I teach about Apache Kafka, the Kafka ecosystem and Kafka Certifications with my focus always on helping my students improve their professional proficiencies. I am also the co-founder of Conduktor: an enterprise Apache Kafka platform & UI to help everyone use Kafka.

Throughout my career in designing and delivering these certifications and courses, I have already taught 1,000,000+ students and gotten 350,000+ reviews!

With Apache Kafka becoming much more than a buzzword out there, I've decided it's time for students to properly learn how to be a Kafka professional. So, let’s kick start the course! You are in good hands!

===============================

This Course Also Comes With:

Lifetime Access to All Future Updates
A responsive instructor in the Q&A Section
Links to interesting articles, and lots of good code to base your next applications onto
Udemy Certificate of Completion Ready for Download

This is the course that could improve your career!

Apache Kafka is a skill in high demand and there are not enough people to fulfill all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies! You can be the next!

I hope to see you inside the course!

=======================

Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:

Learn Kafka for Beginners v2 (great to start)
Kafka Connect Hands On Learning
Kafka Streams for Data Processing
KSQL on ksqlDB - Hands On!
Kafka Cluster Setup & Administration
Confluent Schema Registry & Kafka REST Proxy
Kafka Security (SSL SASL ACL)
Kafka Monitoring and Operations

Happy learning!

Who this course is for:

Developers who would like to learn how to write and run an application that leverages Avro and the Schema Registry
Architects who want to understand the role of the Schema Registry and REST Proxy in an enterprise pipeline
This course is intended for people who fully understand the basics of Kafka
People who want to learn the Confluent Components: Schema Registry and REST Proxy

Course content

9 sections • 62 lectures

Course Introduction 4 lectures

The need of schemas in Kafka Preview 04:10

Short introduction that describes the need of a Schema Registry in Kafka

Course Structure & Objectives Preview 05:57

Learn about the course objectives and how the course is structure

About your instructor Preview 02:20

Learn about Stephane Maarek, your instructor for this course

Architecture for Kafka with the Schema Registry and REST Proxy Preview 02:26

Visit the target architecture with Apache Kafka, the Confluent Schema Registry and the REST Proxy

Code Download 1 lectures

Code Download for the Class Preview 00:14

Download all the code and tools for the course

Avro Schemas 10 lectures

What is Avro? Preview 08:19

Learn what Apache Avro is and how it came to be

Avro Primitive Types Preview 02:16

Learn the Apache Avro primitive types

Avro Record Schema Definition Preview 06:41

Learn how to define a Record Apache Avro Schema

Avro Complex Types Preview 05:46

Learn what are the Complex types for Apache Avro

Practice Exercise: Customer & CustomerAddress Preview 04:51

Practice your newly acquired skills by creating a Customer and CustomerAddress Schema

Avro Logical Types Preview 02:46

Learn what are the possible Apache Avro logical types

The complex case of Decimals Preview 04:01

Learn how to handle decimal numbers in Avro

Example of Avro Schemas on the Web Preview 00:08

Links to various Avro Schemas you can find on the web

Avro Schemas Quiz

Avro Section Summary Preview 00:37

Summary of all the things we have learned so far

Avro in Java 10 lectures

Section Objective Preview 01:16

Understand what we're trying to achieve with Avro in Java

Generic Record in Avro - Hands On Preview 20:10

Learn how to create a GenericRecord in Avro

Specific Record in Avro - Hands On Preview 13:33

Learn how to create a SpecificRecord in Avro

Check-in on now vs later in Kafka Preview 01:43

Check point to understand where we are now vs later when using Kafka

Avro Tools - Hands On Preview 04:15

Hands-On practice with the Avro tools

Reflection in Avro - Hands On Preview 04:44

Learn how to generate an Avro schema using Avro Reflection

Schema Evolution - Theory Preview 07:42

Learn the theory behind schema evolution

Schema Evolution - Hands On Preview 07:33

Get Hands-On with Schema Evolution in Java

Avro in Java Quiz

Avro Java Section Summary Preview 01:51

Section summary to understand how we used Java and Apache Avro library to build Avro objects.

Setup and Launch Kafka 9 lectures

Important information about Kafka setup Preview 01:03

Important information regarding the installation process

Docker on Mac (recent versions) Preview 03:54

Install Docker for Mac if you have a recent Mac machine

Docker Toolbox on Mac (older versions) Preview 05:16

Install Docker Toolbox on Mac if you have an older machine

Docker on Linux (ubuntu as an example) Preview 04:56

Install Docker on Linux

Docker on Windows 10 64bit Preview 03:42

Install Docker on recent Windows 10 64bit

Docker Toolbox on Windows (older versions) Preview 05:37

Install Docker Toolbox on other Windows version

Increasing RAM for Docker - PLEASE READ Preview 01:01

Increase the RAM for Docker

Frequently Asked Questions - Common Issues Preview 02:26

Frequently asked and common issues for Docker and Kafka Setup

Starting Kafka using Docker Compose Preview 04:36

Start our Kafka Cluster using Docker Compose

Confluent Schema Registry and Kafka 11 lectures

Confluent Schema Registry Preview 07:44

Introduction to the Confluent Schema Registry

Kafka Avro Console Producer & Consumer Preview 09:29

Learn how to use the Kafka Avro Console Producer and Consumer

Writing a Kafka Avro Producer in Java Preview 12:58

Learn how to use Java to write your first Avro Producer

Writing a Kafka Avro Consumer in Java Preview 07:40

Learn how to write an Avro consumer using Java

Reminder on Schema Evolution Preview 01:04

Re-visit what Schema Evolution is before we perform it in Java

Writing a V2 Kafka Producer Preview 06:58

Writing a V2 Java Kafka Avro Producer with schema evolution!

Writing a V2 Kafka Consumer Preview 03:45

Writing a V2 Java Kafka Avro Consumer with schema evolution!

Summary on Compatibility Changes Preview 01:15

Summary for Compatibility Changes

Kafka Schema Registry Deep Dive Preview 04:31

Deep Dive to understand how the Schema Registry works behind the scenes

Managing Schemas Efficiently & Section Summary Preview 02:13

Learn how to efficiently manage schemas by having separate repositories for them

Confluent Schema Registry and Kafka Quiz

Confluent REST Proxy 11 lectures

Kafka REST Proxy Introduction and Purpose Preview 03:54

Introduction to understand what is the REST Proxy and what is its purpose

V1 vs V2 APIs Preview 02:18

Understand the difference of the v1 vs the v2 API for the Confluent REST Proxy

Insomnia Setup (REST Client) Preview 01:26

Setup Insomnia on your Machine to have a REST Client

Topic Operations Preview 03:26

Understand how to list topics or view a specific topic's detail using the REST Proxy

Producing in Binary with the Kafka REST Proxy Preview 07:15

Produce Binary data that's base64 encoded using the REST Proxy

Consuming in Binary with the Kafka REST Proxy Preview 05:07

Consume Binary data that's base64 decoded using the REST Proxy

Producing in JSON with the Kafka REST Proxy Preview 03:18

Learn how to use the Confluent REST Proxy to produce data in JSON

Consuming in JSON with the Kafka REST Proxy Preview 03:25

Learn how to use the Confluent REST Proxy to consume data in JSON

Producing in Avro with the Kafka REST Proxy Preview 05:36

Learn how to use the Confluent REST Proxy to produce data in Avro

Consuming in Avro with the Kafka REST Proxy Preview 03:42

Learn how to use the Confluent REST Proxy to consume data in Avro

Confluent REST Proxy Quiz

Annexes 3 lectures

What are Annexes? Preview 00:13

Full Avro End to End: Kafka Producer + Kafka Connect + Kafka Streams Preview 19:00

End to End example for Avro, using Producer, Kafka Connect and Kafka Streams

Kafka REST Proxy Installation and Scaling - Overview Preview 03:39

Learn how to find documentation for the REST Proxy and how to Scale it in Production