Kafka Cluster Setup

Hands-On Training on ZooKeeper Quorum Setup, Kafka Cluster Setup and Administration in AWS.

Last updated 2022-01-10 | 4.7

- Setup a Zookeeper and Kafka cluster on three machines in AWS
- Learn how to deploy Kafka in Production and understand the target architecture for clusters in AWS
- Setup ZooKeeper Cluster
- learn its role for Kafka and usage

What you'll learn

Setup a Zookeeper and Kafka cluster on three machines in AWS
Learn how to deploy Kafka in Production and understand the target architecture for clusters in AWS
Setup ZooKeeper Cluster
learn its role for Kafka and usage
Setup Kafka in Cluster Mode with 3 brokers
including configuration
usage and maintenance
Shutdown and Recover Kafka brokers
to overcome the common Kafka broker problems
Configure Kafka Cluster with production settings and optimisations for better performances based on your workload
Setup web administration tools using Docker: ZooNavigator
Kafka Manager
Confluent Schema Registry
Confluent REST Proxy
Landoop Kafka Topics UI
Administer Kafka using Kafka Manager

* Requirements

* Strong knowledge on Computers
* Servers and Networking
* Strong knowledge of Linux commands
* we will use Ubuntu 16.04
* Good knowledge of AWS
* we will use it as our cloud provider
* Good knowledge of Apache Kafka
* An AWS account and a budget to setup the required machines for the course

Description

Attention all struggling System Administrators, Architects or even Developers! Here is course on how to set up and deploy Apache Kafka in Cluster mode.

Do you want to learn how to setup a Kafka Cluster on multiple servers and stop struggling with it?

Do you want to understand all the steps to perform a Kafka setup and Kafka Deployment in Production?

Have you been trying endlessly to deploy a Kafka Cluster, but haven't had any luck?

If you have answered YES, then you are at the right place. 

Learn from Stephane Maarek, the Best-Selling Instructor on Udemy in Apache Kafka and Big Data! 

This course is designed for System Administrators, Architects and Developers. It includes a step by step guide for installing a ZooKeeper Quorum and a Kafka Cluster. Hands-on training on Kafka Administration is also provided.

This course has Amazon Web Services for our cloud environment, which is used by many companies around the world. AWS Knowledge is a pre-requisite. 

In this course, we will cover what ZooKeeper is, its architecture, its role in Apache Kafka, and the setup, installation and configuration on multiple machines!

This course reveals exactly how your Kafka Cluster on multiple machines should be setup and configured. We will start with understanding the Kafka basics, cluster size and the configuration. We will also have a hands-on learning on AWS Setup, Single Broker Setup, Multi Broker Setup, Testing the Cluster, Kafka Manager (Cluster Management), Demonstrating Kafka Resiliency etc. We will enlighten you on Kafka Performance with respect to I/Os, Network, RAM, CPU, OS (Operating System) etc. You will have hands on learning on how to run Kafka in Production on AWS, how to change a Kafka Broker Configuration and we will also cover the Advanced Kafka Configurations.

Why I should take this course?

  • With over 4 hours of videos and around 50 classes, you will get a great understanding of how to deploy Kafka in production and you will understand the target architecture for clusters in AWS

  • You will learn what ZooKeeper? is, it’s role for Kafka, How to setup ZooKeeper? and its usage

  • You will have a complete understanding Kafka setup including setup, configuration, usage and maintenance etc.

  • You will have a Kafka broker shutdown and recovery demonstration, which will help you to understand how to overcome the Kafka broker problems

  • You will learn Kafka production settings and how to optimise settings for better performance

  • You will learn all the required tool setups such as ZooNavigator, Kafka Manager, Confluent Schema Registry, Confluent REST Proxy, Landoop Kafka Topics UI.

  • You will get hands-on learning on Kafka common Administrative tasks using Kafka Manager

  • You have life-time access to this course and a 30-day satisfaction guaranteed with this course

Overview of the Course Contents -

Section 1 – Course Introduction: In this section, we will have introduction to the course. We will talk about what are the course pre-requisites, course objectives, who are the targeted students for this course. You will also have course material for download in this section!

Section 2 - Target Architecture: In this section, we will see the cloud providers such as AWS and explore other alternatives for AWS.  We will also cover ZooKeeper Quorum Architecture and Kafka Cluster Architecture in this section.   

Section 3 - Zookeeper Quorum Setup: In this sections, we will learn what is ZooKeeper, the Zookeeper role in Kafka Cluster, and ZooKeeper Configuration. We will have hands-on labs on AWS Setup, Zookeeper Single Machine Setup and Zookeeper Quorum Setup.

Section 4 - Kafka Cluster Setup: In this section, we will gain some practical hands-on experience on further AWS Setup, Single Kafka Broker Setup, Multi Kafka Broker Setup etc. Before that we will see Apache Kafka Basics on Kafka Configurations. We will have a demonstration on how to change the Kafka Broker configuration and we will also cover advanced Kafka configurations in this section.

After taking this course, you will become proficient in deploying Kafka in production! You will know all the required setups and installations and Kafka production settings and will be able to set-up Kafka in cluster mode in your own environment!

===============================

Instructor

My name is Stephane Maarek, and I'll be your instructor in this course. I teach about Apache Kafka, the Kafka ecosystem and Kafka Certifications with my focus always on helping my students improve their professional proficiencies. I am also the co-founder of Conduktor: an enterprise Apache Kafka platform & UI to help everyone use Kafka.

Throughout my career in designing and delivering these certifications and courses, I have already taught 1,000,000+ students and gotten 350,000+ reviews!

With Apache Kafka becoming much more than a buzzword out there, I've decided it's time for students to properly learn how to be a Kafka professional. So, let’s kick start the course! You are in good hands!

===============================

This Course Also Comes With:

  • Lifetime Access to All Future Updates

  • A responsive instructor in the Q&A Section

  • Links to interesting articles, and lots of good code to base your next applications onto

  • Udemy Certificate of Completion Ready for Download

This is the course that could improve your career!

Apache Kafka is a skill in high demand and there are not enough people to fulfill all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies! You can be the next!

I hope to see you inside the course!

=======================

Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:

  • Learn Kafka for Beginners v2 (great to start)

  • Kafka Connect Hands On Learning

  • Kafka Streams for Data Processing

  • KSQL on ksqlDB - Hands On!

  • Kafka Cluster Setup & Administration

  • Confluent Schema Registry & Kafka REST Proxy

  • Kafka Security (SSL SASL ACL)

  • Kafka Monitoring and Operations

Happy learning!

Who this course is for:

  • System Administrators or Architects, who would like to learn or setup a Kafka Cluster on multiple servers
  • Developers who want to understand the inner depth of a Kafka setup
  • This course is designed for expert students and should not be taken until all the pre-requisites are met

Course content

5 sections • 48 lectures

Kafka Cluster Setup Introduction Preview 04:46

Cloud Provider & Target Architecture Preview 03:24

Cost of the tutorial Preview 00:38

Important Message Preview 00:37

What is Zookeeper? Preview 04:55

Understand what is Zookeeper and Zookeeper Basics

Zookeeper Role in a Kafka Cluster Preview 02:49

Analyse the role Zookeeper has with regards to Kafka

Zookeeper Quorum Sizing Preview 05:32

Learn how to size a Zookeeper Quorum. 

Zookeeper Configuration Preview 03:14

Learn about Zookeeper Configuration

Hands-On: Zookeeper AWS Setup Preview 11:25

Start configuring our AWS (Amazon Web Services) environment by creating the security groups, and our first EC2 instance

How to SSH Into our Servers? Preview 04:18

Learn how to SSH into our machines. Use Putty if you have a windows machine

Hands-On: Single Zookeeper Machine Setup Preview 12:28

In this lecture we will setup one machine with Zookeeper on it 

Hands-On: Zookeeper Service Setup Preview 06:22

Learn how to setup Zookeeper as a Service on your AWS EC2 machine

Using the Zookeeper Command Line Interface (CLI) Preview 08:13

Learn how to use the Zookeeper Command Line Interface to interact easily with Zookeeper

Hands-On: Zookeeper Quorum setup - Part 1 AWS Preview 08:03

Create two new EC2 instances from a custom AWS AMI

Hands-On: Zookeeper Quorum Setup - Part 2 Zookeeper Preview 15:07

Setup the other two Zookeeper Machines and make sure the quorum is working

Zookeeper Four Letter Words Preview 04:52

Learn about Zookeeper Four Letter Words (4LW) to diagnose issues with Zookeeper

Zookeeper Internal File System Preview 01:59

Learn and understand the files created by Zookeeper

Factors impacting Zookeeper Peformance Preview 01:58

Learn the various factors that can impact heavily Zookeeper's performance

Note: Zookeeper in AWS Preview 01:48

Advice on how to run Zookeeper in AWS

Hands-On: Web tools AWS machine Docker setup Preview 06:20

Setup a new AWS EC2 Machine and install Docker on it, to be able to install our Web tools

Management Tools for Zookeeper & Zoonavigator demo Preview 07:29

Learn about all the various options to manage Zookeeper, and install Zoonavigator to start visualising our Zookeeper cluster

Kafka Basics Preview 01:49

Learn the basics of Apache Kafka

Kafka Cluster Size Discussions Preview 04:26

Understand how to choose the size of a Kafka cluster

Kafka Configuration Preview 08:27

Overview of Kafka Configuration options

Hands-On: Kafka AWS Setup Preview 11:29

Walk through of the required AWS Network and instance changes to setup Kafka

Hands-On: Single Kafka Broker Setup Preview 12:06

Kafka Single Broker Setup, on one instance. We will learn how to start Kafka

Hands-On: Running Kafka Commands Preview 05:11

Learn how to run Kafka Commands against a Kafka Broker

Hands-On: Kafka Multi Broker (Cluster) Setup Preview 10:52

Setup two additional Kafka Brokers to finalise our cluster setup

Hands-On: Testing the Kafka Cluster Preview 05:37

Learn how to test the Kafka Cluster using various Kafka commands

Can I connect to my Kafka cluster? Preview 03:28

Understand how networking works in Kafka and where you can connect to in Kafka

advertised.listeners setting - most important setting Preview 06:11

Learn about the most important Kafka Setting, advertised.listeners, and how it impacts networking

Hands-On: Kafka Manager (Cluster Management) Preview 10:29

Install and start using Kafka Manager for Kafka Cluster Management

Hands-On: Demonstrating Kafka Resiliency Preview 14:25

Demonstrations of Kafka Resiliency by stopping brokers in order, and showing how Kafka Manager and the cluster health changes

Kafka Performance: I/O Preview 01:47

Learn how IO impacts Kafka Performance

Kafka Performance: Network Preview 02:05

Learn how Network impacts Kafka Performance

Kafka Performance: RAM Preview 03:27

Learn how RAM (memory) impacts Kafka performance

Kafka Performance: CPU Preview 01:34

Learn how CPU impacts Kafka Performance

Kafka Performance: OS (Operating System) Preview 01:27

Learn how your OS can impact Kafka Performance

Kafka Performance: Other Preview 01:10

Other notable things to look out with regards to Kafka performance

Running Kafka in Production on AWS Preview 01:36

Learn the subtleties of Running Kafka in production on AWS

Hands-On: How to change a Kafka Broker Configuration Preview 04:01

Hands on lecture to learn how to change a Kafka configuration

Advanced Kafka Configuration Preview 05:43

Learn about the advanced Kafka configuration parameters, that you may want to tweak in production for your cluster

Landoop Kafka Topics UI, Confluent REST Proxy, Confluent Schema Registry Setup Preview 05:37

Kafka Topics UI Setup (alongside Confluent REST Proxy and Confluent Schema Registry)

What's Next? Learning further Preview 01:19

Learn about the next steps in your Kafka in production learning adventure

Congratulations! Preview 01:18

THANK YOU! Preview 01:32

Bonus Lecture - Coupons For My Other Courses - Student Special Offer Preview 01:03

Special discounts for my other courses!!