Kafka Integration

Use this integration when you want to monitor Kafka brokers, topics, and consumers in KloudMate through the OpenTelemetry kafka_metrics receiver running in the KloudMate Agent.

Prerequisites

1. Install and run the KloudMate Agent.
2. Ensure your Kafka brokers are reachable from the host or cluster where the Agent is running.
3. Have your Kafka broker addresses, protocol version, and authentication credentials ready.

Step 1: Open the Collector Configuration

1. Go to Settings → Agents.
2. Find the agent you want to configure and select the three dots.
3. Choose Configure.
4. Go to Collector config.

Step 2: Add the Kafka Metrics Configuration

Add the following configuration. Replace the broker addresses, credentials, and protocol version to match your environment.

• If the agent runs in managed mode, add it to the Custom config override section. See Custom config override.
• If the agent runs in manual mode, add it directly to the collector configuration.

receivers:
  kafka_metrics:
    brokers:
      - kafka.example.com:9095
      - kafka.example.com:9096
      - kafka.example.com:9097
    protocol_version: 2.8.0
    auth:
      plain_text:
        username: <your-username>
        password: <your-password>
    scrapers:
      - brokers
      - topics
      - consumers
    metrics:
      kafka.broker.log_retention_period:
        enabled: true
      kafka.topic.replication_factor:
        enabled: true
      kafka.topic.log_retention_period:
        enabled: true
      kafka.topic.log_retention_size:
        enabled: true
      kafka.topic.min_insync_replicas:
        enabled: true

service:
  pipelines:
    metrics/kafka:
      receivers:
        - kafka_metrics
      processors:
        - batch
      exporters:
        - otlphttp

The scrapers block controls which data is collected. brokers, topics, and consumers are the three available options. The metrics listed under the metrics block are not enabled by default and must be explicitly set to enabled: true to be collected.

Step 3: Save the Configuration

Click Save. The Agent restarts and applies the new configuration.

Step 4: Validate in KloudMate

After the Agent restarts:

• Open Explore in KloudMate.
• Query one of the scraped Kafka metrics, such as kafka.brokers or kafka.topic.partitions.
• Build dashboards or alerts once the data is visible.

Standard Kafka Dashboards

KloudMate provides prebuilt Kafka dashboards through dashboard templates. These dashboards help you visualize broker availability, topic partitions, consumer lag, offsets, and replication health.

To import and start using these templates, follow the steps described in Importing a Dashboard. Alternatively, use From Template when creating a new dashboard to pick from KloudMate’s built-in templates.

Default Kafka Metrics

Kafka Monitoring automatically collects commonly used Kafka metrics when enabled.

Example Metrics

Metric Name	Description
kafka.brokers	Number of brokers in the cluster.
kafka.topic.partitions	Number of partitions in a topic.
kafka.partition.current_offset	Current offset of a topic partition.
kafka.partition.oldest_offset	Oldest available offset of a topic partition.
kafka.partition.replicas	Number of replicas configured for a topic partition.
kafka.partition.replicas_in_sync	Number of in-sync replicas for a topic partition.
kafka.consumer_group.members	Number of members in a consumer group.
kafka.consumer_group.lag	Current approximate lag of a consumer group for a topic partition.
kafka.consumer_group.lag_sum	Current approximate sum of consumer group lag across all topic partitions.
kafka.consumer_group.offset	Current consumer group offset for a topic partition.
kafka.consumer_group.offset_sum	Sum of consumer group offsets across topic partitions.

For the complete metrics list, refer to the metrics reference.