Kafka Monitoring And Operations

  1. Home
  2. Apache-kafka
  3. Kafka Monitoring And Operations

Tags: Apache Kafka

Kafka Monitoring Setup with Prometheus and Grafana, Kafka Operations and Kafka Cluster Upgrades Hands-On. Setup in AWS

Last updated 2022-01-10 | 4.8

- Setup a Multi Broker Kafka Cluster in no-time in AWS (using CloudFormation)
- Setup Administration Tools such as Kafka Manager
- ZooNavigator
- LinkedIn's Kafka Monitor
- Setup Monitoring using Grafana and Prometheus

What you'll learn

Setup a Multi Broker Kafka Cluster in no-time in AWS (using CloudFormation)
Setup Administration Tools such as Kafka Manager
ZooNavigator
LinkedIn's Kafka Monitor
Setup Monitoring using Grafana and Prometheus
Learn how to perform a safe and automated Roll Restart of Kafka Brokers
Update Brokers Configurations in a safe way
Rebalance Partitions in a Kafka Cluster
Increase and Decrease the Replication Factor of Topics
Add a Broker to a Kafka Cluster
Service and Replace a Broker in a Kafka Cluster
Remove a Broker in a Kafka Cluster
Install Command Line Interface (CLI) tools to automate workflows
Upgrade a Kafka Cluster with no downtime

* Requirements

* Kafka Basics + Setup Knowledge
* Linux Knowledge
* Windows / Linux / MacOS X are ok
* Cloud Computing Understanding preferred
* No Knowledge of AWS CloudFormation is required

Description

  • Setup a Multi Broker Kafka Cluster in no-time in AWS (using CloudFormation)
  • Setup Administration Tools such as Kafka Manager, ZooNavigator, LinkedIn's Kafka Monitor
  • Setup Monitoring using Grafana and Prometheus
  • Learn how to perform a safe and automated Roll Restart of Kafka Brokers
  • Update Brokers Configurations in a safe way
  • Rebalance Partitions in a Kafka Cluster
  • Increase and Decrease the Replication Factor of Topics
  • Add a Broker to a Kafka Cluster
  • Service and Replace a Broker in a Kafka Cluster
  • Remove a Broker in a Kafka Cluster
  • Install Command Line Interface (CLI) tools to automate workflows
  • Upgrade a Kafka Cluster with no downtime

Course content

8 sections • 64 lectures

Course Overview and Introduction 2 lectures

Course Overview Preview 07:09

Important Message Preview 01:52

Code Download 1 lectures

Code Download Preview 00:14

Kafka Quick Setup in AWS 8 lectures

What is AWS CloudFormation Preview 01:32

Target Architecture and Setup Preview 04:18

Create AWS Account & Cost of Tutorial Preview 00:22

[Hands-On] Deploy Zookeeper Using CloudFormation Preview 08:15

[Hands-On] Deploy Kafka Using CloudFormation Preview 07:38

[Hands-On] Overview of our Zookeeper Cluster Preview 09:26

[Hands-On] Overview of our Kafka Cluster Preview 05:03

[Hands-On] Stopping the Kafka Cluster Preview 02:49

Kafka Administration Setup 6 lectures

[Hands-On] Overview and Setup of Monitoring Machine Preview 08:10

IMPORTANT: Security Preview 00:25

[Hands-On] Setup Zoonavigator Preview 07:52

[Hands-On] Setup Kafka Manager Preview 08:08

[Hands-On] Setup Kafka Monitor Preview 09:27

Commands Cheat Sheet Preview 00:10

Kafka Monitoring Setup: Grafana + Prometheus 16 lectures

Overview of Monitoring Preview 05:06

[Hands-On] Setting up Prometheus + Kafka Broker 1 Preview 17:31

[Exercise] Broker 2 & 3 Prometheus Setup Preview 00:17

[Solution] Broker 2 & 3 Prometheus Setup Preview 05:20

[Exercise] Zookeeper & Prometheus Setup Preview 00:18

[Solution] Zookeeper & Prometheus Setup Preview 09:40

[Hands-On] Setup Grafana on Administration Server Preview 06:49

[Hands-On] Setup Kafka Dashboard on Grafana Preview 05:43

Important Metrics to monitor Preview 06:04

[Hands-On] Adding Graphs to Grafana Preview 10:15

[Exercise] Adding more graphs to Grafana Preview 00:08

[Solution] Adding more graphs to Grafana Preview 03:01

Find Grafana dashboard in the code Preview 00:03

What about monitoring Kafka Connect, Kafka Streams, etc? Preview 00:26

[Hands-On] Observing Grafana when we kill a broker Preview 05:05

Considerations to make the setup production-ready Preview 01:59

Kafka Operations 20 lectures

[Hands-On] Install Jolokia Agent on Broker 1 Preview 07:54

[Exercise] Install Jolokia on Broker 2 & 3 Preview 00:09

[Solution] Install Jolokia on Broker 2 & 3 Preview 03:38

[Hands-On] Install Rolling Restart Script Preview 14:03

[Hands-On] Run Roll Restart Script Preview 07:21

[Hands-On] Updating Configurations 1st method (rolling restart) Preview 03:45

[Hands-On] Updating Configurations 2nd method (kafka-configs command) Preview 11:18

[Hands-On] Rebalancing Partitions using plain commands and JSON Preview 12:10

[Hands-On] Rebalancing Partitions using Kafka Manager Preview 02:40

[Hands-On] Rebalancing Partitions using LinkedIn tools Preview 08:10

[Hands-On] Increasing Replication Factor using plain commands and JSON Preview 06:38

[Hands-On] Increasing Replication Factor using LinkedIn tools Preview 04:47

[Hands-On] Adding Broker 4 using CloudFormation Preview 09:55

[Exercise] Moving partitions over to Broker 4 using the CLI tools Preview 00:08

[Solution] Moving partitions over to Broker 4 using the CLI tools Preview 04:30

[Hands-On] Replacing a Broker keeping EBS Volume Preview 07:02

[Hands-On] Replacing a Broker losing an EBS Volume Preview 03:41

Removing a Broker Overview Preview 01:00

[Exercise] Remove Broker 4 Preview 00:10

[Solution] Remove Broker 4 Preview 08:01

Kafka Cluster Upgrade 7 lectures

Upgrading a Kafka Cluster Overview Preview 02:19

[Advanced] Zero Copy, Up and Down Conversions Preview 08:06

[Hands-On] Kafka Broker Upgrade Step 1 Preview 04:02

[Hands-On] Kafka Broker Upgrade Step 2 Preview 02:58

[Hands-On] Kafka Broker Upgrade Step 3 Preview 03:03

[Hands-On] Kafka Broker Upgrade Step 4 Preview 01:40

[Hands-On] Kafka Broker Upgrade Step 5 Preview 03:14

Next Steps 4 lectures

Cleaning up AWS Preview 01:17

Next Steps & Congratulations! Preview 02:02

THANK YOU! Preview 01:32

Bonus Lecture: Student Special Coupons for my Other courses Preview 01:03