SMM component architecture

Before you get started with SMM, it is helpful to review this information about SMM component architecture.

The below table outlines key components of SMM as described in the above diagram.


Component Name / Version	Version Supported by SMM	Description	Where it is deployed	Binary Location	Required/Optional
DP Platform	1.2.x	A platform that provides an extensible set of services for HDP and HDF clusters.	DPS_HOST	Customer Support Portal	Required
SMM Application	1.2.0	A DP Platform application that provides a monitoring platform for Kafka clusters.	DPS_HOST	Customer Support Portal	Required
DP Platform Knox	N/A	DP Platform Knox instance is used for SSO between DP Platform and any HDP/HDF Clusters that it is managing. This Knox instances needs to be synced with the same LDAP store as the target clusters.	DPS_HOST	N/A (Part of DP Platform)	Required
HDP or HDF	HDP and/or HDF	SMM supports monitoring Kafka version (2.0.0) that are part of HDP or HDF.	HDP/HDF Nodes	Hortonworks Public Downloads Page	Required (Either HDP or HDF)
SMM REST Server Agent	1.2.0	Provides REST endpoints that power the SMM Application to manage and monitor Kafka clusters. Knox SSO must be enabled for this service. Note that enabling TLS for SMM REST Server Agent is strongly recommended for any production setup.	HDP or HDF Cluster	Customer Support Portal	Required
HDP or HDF Knox	N/A	Provides SSO services from the SMM Application in DP Platform to the SMM REST Server on the HDP/HDF cluster. The logged in user in DP Platform using Knox SSO connects to SMM REST Services, Ambari Services, and other services.	HDP or HDF Cluster	N/A (Part of HDP or HDF)	Required
Ambari	2.7.x	HDP or HDF Cluster must be provisioned by Ambari and SSO must be enabled for Ambari and SMM Rest Server.	HDP/HDF Cluster	Hortonworks Public Downloads Page	Required
Kafka	2.0.0 (installed as part of HDP or HDF)	Kafka 2.0.0 that is installed via HDP or HDF is required for SMM.	HDP or HDF Cluster	N/A (Part of HDP or HDF)	Required
AMS	Part of HDP 3.1.0 and HDF 3.3.x)	Ambari Metrics Server is a time Series and Graph database built on top of HBase. It houses all the time-series Kafka Metrics that power SMM.	HDP/HDF Cluster	N/A (Part of HDP/HDF)	Required
ZooKeeper	Part of HDP and HDF)	ZooKeeper contains information about Kafka metadata.	HDP/HDF Cluster	N/A (Part of HDP/HDF)	Required
AD/LDAP		Identity Provider Store that houses user and group information for users that access the system. DataPlane Platform and HDP or HDF must use the same LDAP instance.	Up to the Organization	Organization specific	Required for Knox SSO integration with HDP or HDF cluster and DP Platform
Kerberos / KDC		System used for strong authentication and identity propagation for both user and services in an HDP or HDF Cluster.	HDP or HDF Cluster	Organization Specific	Optional, but strongly recommended)
Ranger	Part of HDP 3.1 and HDF 3.3.x)	The Kafka Ranger plugin is used by SMM Rest Server to ensure that the logged in user only has access to the topics the user is configured to see based on the configured Ranger policies.	HDP or HDF Cluster	N/A (Part of HDP or HDF)	Optional, but strongly recommended)
Schema Registry	Part of HDF	A central repository that houses schema information including serializers/deserializers for a given Kafka topics. SMM uses SR for the Topic Visualizer capability.	HDF Cluster	N/A (Part of HDF)	Optional, but recommended)
Atlas	Part of HDP	SMM provides links to Kafka entities in Atlas.	HDP cluster	N/A (Part of HDP Cluster)	Optional
Grafana	Part of HDP and HDF)	SMM provides links to broker or topic dashboards in Grafana for more detailed metrics.	HDP or HDF Cluster	N/A (Part of HDP or HDF)	Optional

The below table outlines some of the key integration points between the different SMM components. These integration points are marked in the diagram above with yellow circles.


Integration Point	Description
	HDP and HDF clusters can be registered within DP Platform, if they meet the DP Platform prerequisites. Clusters are registered by providing Ambari Endpoint URL. The DP Platform user that registers a cluster must also be an “Admin” user within that Ambari instance. If these registered clusters have the prerequisite services that SMM requires (see below sections on these prerequisites), each of these clusters can be enabled with the SMM Application. The SMM Application can be used to monitor the Kafka clusters in each of the enabled HDP or HDF clusters.
	This is the integration point between DP Platform Knox and each registered cluster's Knox instance. This integration point is through the configuration of the SSO token topology on the target cluster as documented in the DP Platform Installation Guide. This allows SMM Application to retrieve an SSO token from the target cluster's Knox instance and use that token to make SMM REST Server requests.
	This is the integration point between services in the target cluster (Ambari, SMM Rest Server) and the target instance's Knox service. This integration point allows the service to be SSO enabled. This is configured through the SSO configuration for that service by configuring the Knox's public key within `knox-sso-cert.pem`.
	This integration point is between DP Platform and all the registered clusters to use the same LDAP instance for authentication. Note in this diagram, the DP Platform Knox instance and the Knox instance in Cluster 1 and Cluster 2 all share the same LDAP instance.

The below table outlines the flow of when a SMM Kafka DevOps users named gjvett logs into DP Platform and uses SMM to monitor and manage a Kafka cluster. These integration points are marked in the diagram above with green circles.


	User `gjvett` logs into DP Platform with credentials.
	DP Platform Knox talks to configured LDAP instance to validate the user credentials. If credentials are valid, DP Platform fetches the configured roles for user `gjvett` (roles are managed in DP Platform).
	If `gjvett` has DP Platform Role "Kafka DevOps", then the user sees the SMM Application in DP Platform. User `gjvett` clicks the SMM Application icon and selects a cluster to manage and monitor with SMM.
	For the cluster selected, SMM Application talks to the target cluster's Knox instance to fetch SSO token for user `gjvett` to be able to talk to SMM REST Server Agent on the target cluster.
	SMM Application then makes calls to SMM REST Server Agent as `gjvett` with SSO token stored in cookie header named `hadoop-jwt`.
	SMM REST Server Agent then queries a series of components like Kafka internal consumer group topics, Ambari AMS, Schema Registry, etc to fetch data required for user’s action in the SMM Application.
	The information returned to SMM Application is filtered based on the ACL permissions associated for user `gjvett` (only information on topics that user `gjvett` has permission to see is returned.)