Debugging

If you experience issues when you tune your system, you can use several tools to determine the source of the issues.

Before using any of the tools, set the following environment variables for your cluster. The following example illustrates how we would configure the environment variables:

export HDP_HOME=/usr/hdp/current
export KAFKA_HOME=$HDP_HOME/kafka-broker
export STORM_UI=http://$STORM_HOST:8744
export ELASTIC=http://$ELASTICSEARCH_HOST:9200
export ZOOKEEPER=$ZOOKEEPER_HOST:2181
export METRON_VERSION=1.4.2
export METRON_HOME=/usr/hcp/${METRON_VERSION}

Note that the output from Storm will be a flattened JSON file. To print the file for readability, pipe it through a JSON formatter. For example:

[some Storm curl command] | python -m json.tool

Getting Storm Configuration Details

To get full details about your running topologies, see the Storm view. For information on using the Storm View, see Using Storm View. If you have never used Apache Ambari Views, see Creating View Instances to get started.

Alternatively, Storm has a useful REST API you can use to get full details about your running topologies. See Storm's REST API docs for more details.

# get Storm cluster summary info including version
curl -XGET ${STORM_UI}'/api/v1/cluster/summary'

# get overall Storm cluster configuration
curl -XGET ${STORM_UI}'/api/v1/cluster/configuration'

# get list of topologies and brief summary detail
curl -XGET ${STORM_UI}'/api/v1/topology/summary'

# get all topology runtime settings. Plugin the ID for your topology, which you can get from the topology summary command or from the Storm UI. Passing sys=1 will also return system stats.
curl -XGET ${STORM_UI}'/api/v1/topology/:id?sys=1'

Getting Kafka Configuration Details

# Get list of Kafka topics
${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --list

# Get Kafka topic details - plugin the desired topic name in place of "enrichments"
${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --topic enrichments --describe

Getting Metron Topology ZooKeeper Configuration

# Provides a full listing of all Metron parser, enrichment, and indexing topology configuration
$METRON_HOME/bin/zk_load_configs.sh -m DUMP -z $ZOOKEEPER

​Debugging

Debugging