카프카 클러스터 모니터링 및 관리 툴 비교

What are the best tools engineers can use to observe data flows, track key metrics, and troubleshoot issues in Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform that enables organizations to implement and handle high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. It is used by thousands of companies, 80% of which are Fortune 100 companies.

Though Apache Kafka is the go-to service for scenarios requiring real-time data processing and application activity tracking, the monitoring and management of clusters in Apache Kafka often pose a challenge. To make these tasks more efficient and transparent, you may need third-party, open-source, or commercial graphical tools that offer additional administrative and monitoring features.

This article provides an overview of such UI tools, including:

1. AKHQ
2. Kowl
3. Kafdrop
4. UI for Apache Kafka
5. Lenses
6. CMAK
7. Confluent CC
8. Conduktor

But first, let’s delve a bit into the problems of observability and monitoring in Apache Kafka.

Observability and Monitoring in Apache Kafka

Apache Kafka is a critical service for any data-driven organization. But handling vanilla Apache Kafka clusters can be quite painful. Kafka clusters are difficult to set up, tricky to scale, and expensive to maintain. Most importantly, they are error-prone and complicated to observe, which may cause various business-critical issues in data streams and prevent engineers from tracking and solving them.

To reduce errors and avoid critical issues, it is important for organizations to foolproof their Apache Kafka clusters by adding a robust observability component. Better observability can help to:

• Troubleshoot data streams much faster
• Improve collaboration among engineers by having a shared understanding of the data streams they manage through metadata.
• Discover sensitive data within data streams more easily, to meet compliance requirements.
• Clean data more quickly and efficiently. Fewer incorrect dashboards mean happier customers.

The challenge with cluster observability is that Apache Kafka comes with CLI tools for all necessary administrative tasks. But because they are not integrated into a single service, you have to run different tools separately for different tasks, which causes inconvenience and time loss. The problem can escalate quickly as your clusters grow in size, or if you have multiple clusters, simply because you lack the ability to properly observe them.

This brings us back to UI monitoring tools that can help organizations simplify and accelerate development, minimize time to resolution, and speed up the reporting process, to improve operational efficiency within and between engineering teams.

Top 8 UI Monitoring Tools for Apache Kafka Clusters

To begin with, a quick comparison of the Apache Kafka cluster monitoring tools.

 

Image by Author

AKHQ

• GitHub: https://github.com/tchiotludo/akhq
• License: Apache 2
• Availability: Free
• Pros: A lot of useful features
• Cons: Bad UI; Lacks KSQL integration; Partial support of Protobuf schema registry

AKHQ (previously known as KafkaHQ) is a Kafka GUI for Apache Kafka that gives engineering teams the ability to search and explore data in a unified console. With AKHQ, developers and DevOps can manage topics, topic data, consumer groups, schema registry, connectivity, and more.

 

Image by Author

AKHQ offers tons of useful features, including multi-cluster management, message browsing, live tailing, authentication, authorization, read-only mode, schema registry, and Kafka Connect management. It supports Avro and is compatible with LDAP and RBAC.

But AKHQ’s UI is not the most convenient. You will definitely need to allocate some time to scale the learning curve.

On top of that, AKHQ does not have KSQL integration and provides only partial support of the Protobuf schema registry. Nor does it work with Dynamic Topic Configuration, Partition Increase, Replica Change, Kafka Streams Topologies, or JMX metrics visualization and charts.

Those who are looking to use AKHQ as part of their data streaming solutions on AWS should know that it does not support AWS Identity and Access Management (IAM) Access Control for Amazon MSK.

Kowl

• GitHub: https://github.com/cloudhut/kowl
• License: Apache 2
• Availability: Partially paid
• Pros: Good UI
• Cons: Lacks many features

Kowl (previously known as Kafka Owl) is a web application designed to help developers explore messages in Apache Kafka clusters and gain better insights into what is actually happening in these clusters.

 

Image by Author

The biggest advantage of Kowl is its fantastic UI. It is convenient, user-friendly, and quite straightforward to use. It does not come with a lot of features, though.

For instance, Kowl offers message browsing, live tailing, and support of Protobuf, Avro, and Amazon MSK IAM, but login system (Google, GitHub, Okta) and RBAC permissions with group syncing is available only with a paid Kowl Business plan.

Kowl is also missing such features as multi-cluster management, dynamic topic configuration, partition increase, replica change, Kafka Connect management, schema registry, KSQL integration, Kafka Streams topologies, read-only mode, and visualization and charts for JMX metrics. If these were included in the package, Kowl would be preferable to any other tool.

Kafdrop

• GitHub: https://github.com/obsidiandynamics/kafdrop
• License: Apache 2
• Availability: Free
• Pros: Nice community
• Cons: Average UI; Lacks a lot of features

Kafdrop is a web UI for viewing Apache Kafka topics and browsing consumer groups. The tool enables developers to more easily display and handle cluster information such as brokers, topics, partitions, and consumers. It also allows for viewing messages.

 

Image by Author

For the most part, Kafdrop is a pretty average tool. Its UI is not spectacular, and it lacks a lot of features. It does allow you to view Kafka brokers and consumer groups, create and view topics, browse messages, and monitor ACLs. It also provides support for Azure Event Hubs. But what about other useful features like live tailing, schema registry, or read-only mode?

The good news is that Kafdrop is highly rated on GitHub, and if you are looking for a helpful and immersive community, this may well be the tool for you.

UI for Apache Kafka

• GitHub: https://github.com/provectus/kafka-ui
• License: Apache 2
• Availability: Free
• Pros: Good UI; Flexible; Lots of features
• Cons: Development in progress

UI for Apache Kafka is a free open-source web service that offers developers a clear UI for handling Apache Kafka clusters. It enables developers to monitor data flows and find and troubleshoot issues in data while delivering optimal performance. The lightweight dashboard makes it easy to keep track of key metrics of Apache Kafka clusters, including Brokers, Topics, Partitions, Production, and Consumption.

 

Image by Author

UI for Apache Kafka stands out from the crowd thanks to its convenient UI, free availability, and sheer number of features. It boasts such capabilities as:

• Browse Messages — browse messages with Avro, Protobuf, JSON, and plain text encoding
• View Consumer Groups — view per-partition parked offsets, and combined and per-partition lag
• Configurable Authentification — secure your installation with optional Github/Gitlab/Google OAuth 2.0
• View Kafka Brokers — view topic and partition assignments, and controller status
• View Kafka Topics — view partition count, replication status, and custom configuration
• Multi-Cluster Management — monitor and manage all your clusters in one place
• Dynamic Topic Configuration — create and configure new topics with dynamic configuration

Provectus, the AI consultancy that is designing and building the tool, claims that it will add more features, including live tailing, KSQL integration, Kafka Streams topologies, and JMX metrics visualization and charts, as early as the end of August.

Lenses

• GitHub: https://github.com/lensesio
• License: BSL
• Availability: Free
• Pros: Awesome with fast-kafka-dev and for local development
• Cons: Lacks many features

Lenses positions itself as a DataOps platform for real-time applications and data operations for Apache Kafka and Kubernetes. It can help engineers make data more usable and secure, and eliminate data silos. Lenses appear to be the highest-rated product for real-time stream analytics.

But Lenses is arguably a fairly average tool. It makes perfect sense to use Lenses with fast-kafka-dev. It is also good for local development. However, it lacks certain features; multi-cluster management, message browsing, and avro support are simply not enough to make it work for many tasks. Offering Kafka Connect management as a separate service does not help, either.

 

Image by Author

But if you are content without a lot of features, Lenses’s UI will absolutely fit the bill. It is a truly stunning tool that is very sleek and intuitive.

CMAK

• GitHub: https://github.com/yahoo/CMAK
• License: Apache 2
• Availability: Free
• Pros: Great for partition reassignment; Ops tool
• Cons: Limited to Ops tasks

CMAK (previously known as Kafka Manager) is a comprehensive tool that enables engineers to manage Apache Kafka clusters for various Ops tasks.

 

Image by Author

CMAK boasts a good and fairly straightforward UI. Though it does not provide a lot of features, multi-cluster management, dynamic topic configuration, partition creation, and replica change will cover most of your tasks.

For the most part, CMAK is primarily an Ops tool. It is also really good at partition reassignment.

Confluent CC

• GitHub: https://github.com/confluentinc
• License: Paid
• Availability: Paid
• Pros: Part of Confluent Enterprise
• Cons: Part of Confluent Enterprise

Confluent Control Center is a web-based user interface that allows developers and operators to manage and monitor Apache Kafka clusters, including checking cluster health, observing and controlling messages, topics, and Schema Registry. It can also be used to develop and run ksqlDB queries.

The important thing about Confluent CC is that it is offered as part of Confluent Enterprise, meaning it is a paid service only. It comes with lots of features and a really good user interface. If you don’t mind being locked into the Confluent ecosystem, this UI tool will more than meet your needs.

Overall, Confluent CCS is more than just an ordinary topic checking tool. Its features are superb, and all of them work perfectly, without any glitches.

Conduktor

• GitHub: https://github.com/conduktor
• License: Paid
• Availability: Paid
• Pros: Tons of features
• Cons: Desktop tool

Conduktor is a desktop client for Apache Kafka that provides engineers with a user-friendly interface to work with the Kafka ecosystem. The client is native to Windows, Linux, and Mac. It can handle any type of Apache Kafka cluster and boasts every feature you could ask for.

Image by Author

But Conduktor may not be as convenient as other UI tools in this list, because it is a desktop application. If you are OK with that, Conduktor can be a viable alternative to Confluent CC.

Conclusion

Having the right UI tools for monitoring and management of Apache Kafka clusters is key to cluster health. With a simple and convenient user interface, you can observe data flows, track metrics, and troubleshoot issues more efficiently, without having to use dozens of additional CLI tools. This results in fewer bottlenecks, faster reporting, and more cost-efficient development.

This article offers my perspective on major UI tools for monitoring and management of clusters in Apache Kafka. I did my best to be objective, but the community is sure to have something to add.

Feel free to share your feedback and opinions about these UI tools for Apache Kafka, and my overview of them, in the comment section.