Prometheus properties configuration

Learn the properties that you need to configure in the prometheus.yml file before you start using the Prometheus metric store for Streams Messaging Manager (SMM).

Configure the following properties in the prometheus.yml file:

Set the scrape_interval property value to 60 seconds in the prometheus.yml file.
```
scrape_interval: 60s
```

Prometheus automatically assigns the instance label to metrics. However, in case partition leader change reassignment happens, it means an excessive amount of metrics being created. This property is recommended for large clusters.

    metric_relabel_configs:
    - source_labels: [__name__]
      regex: ^(broker_producer_messagesinpersec_total|topic_partition_messagesinpersec_total|topic_partition_bytesinpersec_total|topic_partition_bytesoutpersec_total)$
      target_label: instance
      replacement: 'no-instance'

Set Kafka host and metrics port values.
```
['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042']
```
You can find the metric port used by Kafka in Cloudera Manager > Kafka service > Configuration > HTTP Metric Report Port.
Set Kafka Connect host (deployed on same hosts as Kafka) and the metrics port values.
```
['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087']
```
The metrics port used by Kafka Connect is set in Cloudera Manager > Kafka service > Configuration > Secure Jetty Metrics Port and Jetty Metrics Port. Use Secure Jetty Metrics Port if the Kafka Connect metrics reporter uses TLS/SSL, otherwise, use Jetty Metrics Port. The value of 28087 in this example is the default secure port.
Set Prometheus node exporter host (deployed on same hosts as Kafka) and the Prometheus node exporter metrics port values.
```
['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100']
```
update=true parameter should be only used by Prometheus. Querying Kafka Prometheus endpoint with this flag set to true updates the internal metrics cache for stateful metrics.
```
update: ['true']
```
If Basic Authentication is enabled for the Kafka Connect metrics reporter, add the following parameters:
```
basic_auth:
  username: 'email@username.me'
  password: 'password'
```
Ensure that you specify the username and password that is configured for the Kafka Connect metrics reporter. You can find the username and password in Cloudera Manager > Kafka service > Configuration > Jetty Metrics User Name and Jetty Metrics Password.

Configuration example

The following is an example of a Prometheus configuration YAML. You can use this example as a template and make changes as necessary. Ensure that you replace host and port values as well as the Basic Authentication credentials with your own values. This example uses the default ports.

# my global config
global:
  scrape_interval: 60s 
  scrape_timeout: 55s

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'kafka'
    
    metrics_path: '/api/prometheus-metrics'

    params:
      update: ['true']
    
    static_configs:
    - targets: ['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042']
    metric_relabel_configs:
    - source_labels: [__name__]
      regex: ^(broker_producer_messagesinpersec_total|topic_partition_messagesinpersec_total|topic_partition_bytesinpersec_total|topic_partition_bytesoutpersec_total)$
      target_label: instance
      replacement: 'no-instance'

  - job_name: 'kafka_connect'

    metrics_path: '/api/prometheus-metrics'

    basic_auth:
      username: 'email@username.me'
      password: 'password'

    static_configs:
    - targets: ['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087']

  - job_name: 'system_metrics'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100']

Where:

['luigi-1.luigi.root.hwx.site:9100','luigi-2.luigi.root.hwx.site:9100','luigi-3.luigi.root.hwx.site:9100'] = Kafka host + Node exporter metrics port
['luigi-1.luigi.root.hwx.site:24042','luigi-2.luigi.root.hwx.site:24042','luigi-3.luigi.root.hwx.site:24042'] = Kafka host + HTTP Metric Report Port
['luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087','luigi-1.luigi.root.hwx.site:28087'] = Kafka Connect host + Secure Jetty Metrics Port