Hortonworks Cybersecurity Platform
Also available as:
PDF
loading table of contents...

Create Multiple Parsers on One Topology

You can specify multiple parsers to run on one aggregated Storm topology to conserve resources. However, for performance reasons, you should group multiple parsers that have similar velocity or data flow and perform functions with similar complexity.

  1. Use the CLI to create multiple parsers that you want to specify on a single Storm topology.
    Note
    Note
    All parsers must use the same error topic.
    Refer to Create a Parser for Your New Data Source by Using the CLI.
  2. Deploy the new parser topologies to the cluster:
    1. Log in to the host that has Metron installed as root user.
    2. Deploy the new parsers you want to specify onto one topology:
      $METRON_HOME/bin/start_parser_topology.sh -k $KAFKA_HOST:6667 -z $ZOOKEEPER_HOST:2181 -s $DATASOURCE_ONE,$DATASOURCE_TWO
      For example:
      $METRON_HOME/bin/start_parser_topology.sh -z $ZOOKEEPER_HOST:2181 -s bro,yaf
    3. If you want to override parser parameters, you can add the parameter and its value to the deployment command.
      For a list of parser parameters, see Create a Parser for Your New Data Source by Using the CLI.
      For example:
      $METRON_HOME/bin/start_parser_topology.sh -z $ZOOKEEPER_HOST:2181 -s bro,yaf -spoutNumTasks 2,3 -parserParallelism 2 -parserNumTasks 5
      This command will create a toplogy with the following parameters:
      • Bro - spout number of tasks = 2
      • YAF - spout number of tasks = 3
      • YAF - paser parallelism = 2
      • YAF - parser number of tasks = 5
    4. Use the Apache Storm UI to verify that the new topology is listed and that it has no errors.
    This new data source processor topology ingests from each $DATASOURCE Kafka topic that you created earlier and then parses the event with the HCP Grok framework using the Grok pattern defined earlier.