How to Configure Kafka Monitoring Using Prometheus Mode

Note: This document explains how to configure Apache Kafka monitoring through Prometheus integration in Applications Manager.

1. Overview of Monitoring Components

• Kafka Broker: Generates JMX metrics.
• JMX Exporter: Converts Kafka JMX metrics into Prometheus-compatible metrics.
• Prometheus Server: Scrapes and stores metrics exposed by JMX Exporter.
• Applications Manager: Discovers and monitors Kafka metrics through Prometheus APIs.

Monitoring Flow:

Kafka Broker
     |
     v
JMX Exporter (7071)
     |
     v
Prometheus Server (9090)
     |
     v
Applications Manager

2. Prerequisites

• Apache Kafka installed and running.
• ZooKeeper running (for Kafka versions using ZooKeeper mode).
• Java installed and configured.
• Prometheus JMX Exporter JAR downloaded.
• Prometheus Server downloaded.
• Applications Manager build supporting Prometheus Kafka monitoring.

3. Configure Kafka JMX

Enable JMX while starting Kafka.

Example:

set JMX_PORT=1304

set KAFKA_OPTS=-Dcom.sun.management.jmxremote ^
-Dcom.sun.management.jmxremote.port=1304 ^
-Dcom.sun.management.jmxremote.rmi.port=1304 ^
-Dcom.sun.management.jmxremote.authenticate=false ^
-Dcom.sun.management.jmxremote.ssl=false ^
-Djava.rmi.server.hostname=<Kafka_Hostname> ^
-Dcom.sun.management.jmxremote.local.only=false

Start Kafka:

kafka-server-start.bat ..\..\config\server.properties

Verify JMX port:

netstat -ano | findstr 1304

4. Configure Prometheus JMX Exporter

Step 1: Download JMX Exporter

Download the Prometheus JMX Exporter Java Agent JAR and place it in a directory such as:

C:\jmx

Step 2: Create Kafka Rules File

Create the configuration file:

C:\jmx\kafka.yml

Use the provided Kafka rules file (attached separately) which contains Kafka-specific metric mappings and naming conventions.

Step 3: Configure Kafka to Load JMX Exporter

Add the following Java Agent configuration:

set KAFKA_OPTS=-javaagent:C:\jmx\jmx_prometheus_javaagent.jar=7071:C:\jmx\kafka.yml

Restart Kafka.

Step 4: Verify Metrics Endpoint

Open:

http://<KafkaHost>:7071/metrics

You should see Prometheus-formatted metrics beginning with:

# HELP
# TYPE

5. Install Prometheus Server if not available

Step 1: Download Prometheus

Download Prometheus Windows package from:

https://prometheus.io/download/

Extract to:

C:\prometheus

Step 2: Configure Prometheus

Edit:

C:\prometheus\prometheus.yml

Add Kafka scrape configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'kafka'
    static_configs:
      - targets:
          - '<KafkaHost>:7071'

Step 3: Start Prometheus

prometheus.exe --config.file=prometheus.yml

6. Verify Prometheus Configuration

Verify Prometheus UI

Open:

http://<PrometheusHost>:9090

Verify Kafka Target

Navigate to:

Status → Targets

The Kafka target should appear as:

UP

Verify API Response

Open:

http://<PrometheusHost>:9090/api/v1/query?query=up

A valid JSON response should be returned.

7. Add Prometheus Monitor in Applications Manager

Navigate to:

New Monitor → Prometheus

Configure:

Important:

• Do not configure Applications Manager to connect directly to port 7071.
• Port 7071 belongs to JMX Exporter and exposes only Prometheus text metrics.
• Applications Manager requires the Prometheus API endpoint available on port 9090.

8. Resynchronize Kafka Monitor

After configuring Prometheus integration:

1. Open the Prometheus monitor.
2. Click Resync.
3. Kafka instances should be discovered automatically.

9. Troubleshooting

MalformedJsonException

Cause:

Applications Manager configured with port 7071.

Resolution:

Configure Applications Manager to connect to Prometheus on port 9090 instead of JMX Exporter.

JMX Exporter Startup Failure

Error:

FileNotFoundException: kafka.yml

Resolution:

• Verify kafka.yml exists.
• Verify the Java Agent path is correct.
• Verify file extensions are not hidden (e.g., kafka.yml.txt).

Kafka Not Appearing in Prometheus Targets

• Verify Kafka is running.
• Verify JMX Exporter endpoint is accessible.
• Verify target configuration in prometheus.yml.
• Verify firewall access to port 7071.

10. Validation Checklist