Create Multiple Parsers on One Topology

You can specify multiple parsers to run on one aggregated Storm topology to conserve resources. However, for performance reasons, you should group multiple parsers that have similar velocity or data flow and perform functions with similar complexity.

1. Use the CLI to create multiple parsers that you want to specify on a single Storm topology.
   Refer to Create a Parser for Your New Data Source by Using the CLI.
2. Deploy the new parser topologies to the cluster:
   1. Log in to the host that has Metron installed as root user.
   2. Deploy the new parsers you want to specify onto one topology:

      $METRON_HOME/bin/start_parser_topology.sh -k $KAFKA_HOST:6667 -z $ZOOKEEPER_HOST:2181 -s $DATASOURCE_ONE,$DATASOURCE_TWO

      Note

      If your parser name contains a hyphen, you must enclose the parser name in single quotes ('). If you do not enclose a hyphenated parser name in single quotes, Ambari will assume each word or character in the hyphenated parser name is a separate parser. For example, Ambari interprets sapower-windows-x-json,bro as seven parsers instead of one. Even the hyphen is considered a parser. For clarity and consistency, we recommend enclosing all parser names in single quotes when you deploy the new parsers onto a topology.

      For example:

      $METRON_HOME/bin/start_parser_topology.sh -z $ZOOKEEPER_HOST:2181 -s 'bro-source1','yaf'

   3. If you want to override parser parameters, you can add the parameter and its value to the deployment command.
      For a list of parser parameters, see Create a Parser for Your New Data Source by Using the CLI.
      For example:

      $METRON_HOME/bin/start_parser_topology.sh -z $ZOOKEEPER_HOST:2181 -s 'bro-source1','yaf' -spoutNumTasks 2,3 -parserParallelism 2 -parserNumTasks 5

      This command will create a topology with the following parameters:
      • Bro - spout number of tasks = 2
      • YAF - spout number of tasks = 3
      • YAF - parser parallelism = 2
      • YAF - parser number of tasks = 5
   4. Use the Apache Storm UI to verify that the new topology is listed and that it has no errors.
   This new data source processor topology ingests from each $DATASOURCE Kafka topic that you created earlier and then parses the event with the HCP Grok framework using the Grok pattern defined earlier.