The Ins and Outs of Apache Kafka, Part 3
Brett Orr
General Manager Bigmate

About This Workshop

In this workshop, we'll take a look at simple deployment options for setting up and bringing up a basic Kafka cluster. We'll also dive into monitoring solutions for a Kafka Cluster and discuss important metrics and configuring alerts and notifications.

The breakdown of the workshop will be as follows:

Section I:
We'll look at a dockerized solution and create a Dockerfile that deploys a docker container for Kafka. We'll look at simple Dashboard provided by Lenses.We'll also walk through sample terraform and ansible scripts that deploy a Kafka cluster as EC2 instances in AWS. Finally we'll configure Kafka and zookeeper settings for systemd service configurations and look at example Kafka-to-S3 connector that is deployed in AWS that integrates with S3 Buckets.

Section II:
For an overall monitoring solution, we'll start by looking at Kafka logging and discuss the different logs that are configured with a simple deployment. We'll look at broker logs, zookeeper logs, server logs, connect logs, and discuss how these logs can be made available to support teams. Additional topics covered in this section include:
  • Monitoring each component of a Kafka Cluster: service, application, data flows, brokers, zookeeper, schema reg, connect, network monitoring, System - cpu, disk space, memory
  • JMX_Exporter and Kafka metrics reported by JMX_Exporter
  • Sample Kafka dashboards provided by Grafana
  • Splunk Connect for Kafka - a sink connector that allows a Splunk software administrator to subscribe to a Kafka topic and stream the data to the Splunk HTTP event collector
  • Elasticsearch Metricbeat Kafka module and how it integrates with Jolokia

Section III:
We'll look at configuring Kafka related alerts related to: InSync Replicas, Partitions, Broker Connection, Zookeeper Connection, Health Status, latency, bandwidth, throughput, consumer lag, and preventing loss of messages in production. Finally we'll wrap up by discussing notification options including email, Slack, Service-Now, and PagerDuty.

Swati Kher
Senior Consultant

© Instaclustr Copyright
Apache Cassandra®, Apache Kafka®, Apache Spark™, Apache Kafka®, and Apache ZooKeeper™ are trademarks of the Apache Software Foundation. Elasticsearch™ and Kibana™ are trademarks for Elasticsearch BV. Kubernetes® is a registered trademark of the Linux Foundation. OpenSearch is a registered trademark of Amazon Web Services. Postgres®, PostgreSQL® and the Slonik Logo are trademarks or registered trademarks of the PostgreSQL Community Association of Canada, and used with their permission. Redis™ is a trademark of Redis Labs Ltd. *Any rights therein are reserved to Redis Labs Ltd. Any use by Instaclustr Pty Limited is for referential purposes only and does not indicate any sponsorship, endorsement or affiliation between Redis and Instaclustr Pty Limited. All product and service names used in this website are for identification purposes only and do not imply endorsement.