Installing Streams Messaging Manager
Also available as:
PDF

SMM component architecture

Before you get started with SMM, it is helpful to review this information about SMM component architecture.

The below table outlines key components of SMM as described in the above diagram.

Component Name / Version Version Supported by SMM Description Where it is deployed Binary Location Required/Optional
DP Platform 1.2.0 A platform that provides an extensible set of services for HDP and HDF clusters. DPS_HOST Customer Support Portal Required
SMM Application 1.1.0 A DP Platform application that provides a monitoring platform for Kafka clusters. DPS_HOST Customer Support Portal Required
DP Platform Knox N/A DP Platform Knox instance is used for SSO between DP Platform and any HDP/HDF Clusters that it is managing. This Knox instances needs to be synced with the same LDAP store as the target clusters. DPS_HOST N/A (Part of DP Platform) Required
HDP / HDF

HDP and/or HDF

SMM supports monitoring Kafka version (1.1.1) that are part of HDP or HDF. HDP/HDF Nodes Hortonworks Public Downloads Page Required (Either HDP or HDF)
SMM REST Server Agent 1.1.0 Provides REST endpoints that power the SMM Application to manage and monitor Kafka clusters. Knox SSO must be enabled for this service. Note that enabling TLS for SMM REST Server Agent is strongly recommended for any production setup. HDP or HDF Cluster Customer Support Portal Required
HDP/HDF Knox N/A Provides SSO services from the SMM Application in DP Platform to the SMM REST Server on the HDP/HDF cluster. The logged in user in DP Platform using Knox SSO connects to SMM REST Services, Ambari Services, and other services. HDP or HDF Cluster N/A (Part of HDP/HDF) Required
Ambari 2.7.1 HDP or HDF Cluster must be provisioned by Ambari and SSO must be enabled for Ambari and SMM Rest Server. HDP/HDF Cluster Hortonworks Public Downloads Page Required
Kafka 1.1.1 (installed as part of HDP / HDF) Kafka 1.1.1 that is installed via HDP or HDF is required for SMM. HDP/HDF Cluster N/A (Part of HDP/HDF) Required
AMS Part of HDP 3.0 and HDF 3.2) Ambari Metrics Server is a time Series and Graph database built on top of HBase. It houses all the time-series Kafka Metrics that power SMM. HDP/HDF Cluster N/A (Part of HDP/HDF) Required
Zookeeper Part of HDP and HDF) ZooKeeper contains information about Kafka metadata. HDP/HDF Cluster N/A (Part of HDP/HDF) Required
AD/LDAP Identity Provider Store that houses user and group information for users that access the system. Data Plane and HDP/HDF must use the same LDAP instance. Up to the Organization Organization specific Required for Knox SSO integration with HDP/HDF cluster and DP Platform
Kerberos / KDC System used for strong authentication and identity propagation for both user and services in an HDP/HDF Cluster. HDP/HDF Cluster Organization Specific Optional, but strongly recommended)
Ranger Part of HDP 3.0 and HDF 3.2) The Kafka Ranger plugin is used by SMM Rest Server to ensure that the logged in user only has access to the topics the user is configured to see based on the configured Ranger policies. HDP/HDF Cluster N/A (Part of HDP/HDF) Optional, but strongly recommended)
Schema Registry Part of HDF A central repository that houses schema information including serializers/deserializers for a given Kafka topics. SMM uses SR for the Topic Visualizer capability. HDF Cluster N/A (Part of HDF)

Optional, but recommended)

Atlas Part of HDP SMM provides links to Kafka entities in Atlas. HDP cluster N/A (Part of HDP Cluster) Optional
Grafana Part of HDP and HDF) SMM provides links to broker or topic dashboards in Grafana for more detailed metrics. HDP/HDF Cluster N/A (Part of HDP/HDF) Optional

The below table outlines some of the key integration points between the different SMM components. These integration points are marked in the diagram above with yellow circles.

Integration Point Description
  • HDP and HDF clusters can be registered within DP Platform, if they meet the DP Platform prerequisites. Clusters are registered by providing Ambari Endpoint URL.

  • The DP Platform user that registers a cluster must also be an “Admin” user within that Ambari instance.

  • If these registered clusters have the prerequisite services that SMM requires (see below sections on these prerequisites), each of these clusters can be enabled with the SMM Application.

  • The SMM Application can be used to monitor the Kafka clusters in each of the enabled HDP/HDF clusters.

  • This is the integration point between DP Platform Knox and each registered cluster's Knox instance.

  • This integration point is through the configuration of the SSO token topology on the target cluster as documented in the DP Platform Installation Guide.

  • This allows SMM Application to retrieve an SSO token from the target cluster’s Knox instance and use that token to make SMM REST Server requests.

  • This is the integration point between services in the target cluster (Ambari, SMM Rest Server) and the target instance’s Knox service.

  • This integration point allows the service to be SSO enabled.

  • This is configured through the SSO configuration for that service by configuring the Knox’s public key within knox-sso-cert.pem.

  • This integration point is between DP Platform and all the registered clusters to use the same LDAP instance for authentication.

  • Note in this diagram, the DP Platform Knox instance and the Knox instance in Cluster 1 and Cluster 2 all share the same LDAP instance.

The below table outlines the flow of when a SMM Kafka DevOps users named gjvett logs into DP Platform and uses SMM to monitor and manage a Kafka cluster. These integration points are marked in the diagram above with green circles.

User gjvett logs into DP Platform with credentials.

  • DP Platform Knox talks to configured LDAP instance to validate the user credentials.

  • If credentials are valid, DP Platform fetches the configured roles for user gjvett (roles are managed in DP Platform).

  • If gjvett has DP Platform Role “Kafka DevOps”, then the user sees the SMM Application in DP Platform.

  • User gjvett clicks the SMM Application icon and selects a cluster to manage and monitor with SMM.

For the cluster selected, SMM Application talks to the target cluster’s Knox instance to fetch SSO token for user gjvett to be able to talk to SMM REST Server Agent on the target cluster.

SMM Application then makes calls to SMM REST Server Agent as gjvett with SSO token stored in cookie header named hadoop-jwt.

SMM REST Server Agent then queries a series of components like Kafka internal consumer group topics, Ambari AMS, Schema Registry, etc to fetch data required for user’s action in the SMM Application.

The information returned to SMM Application is filtered based on the ACL permissions associated for user gjvett (only information on topics that user gjvett has permission to see is returned.)