Create Multiple Parsers on One Topology
You can specify multiple parsers to run on one aggregated Storm topology to conserve resources. However, for performance reasons, you should group multiple parsers that have similar velocity or data flow and perform functions with similar complexity.
Use the CLI to create multiple parsers that you want to specify on a single Storm
NoteAll parsers must use the same error topic.Refer to Create a Parser for Your New Data Source by Using the CLI.
Deploy the new parser topologies to the cluster:
This new data source processor topology ingests from each $DATASOURCE Kafka topic that you created earlier and then parses the event with the HCP Grok framework using the Grok pattern defined earlier.
- Log in to the host that has Metron installed as root user.
Deploy the new parsers you want to specify onto one topology:
$METRON_HOME/bin/start_parser_topology.sh -k $KAFKA_HOST:6667 -z $ZOOKEEPER_HOST:2181 -s $DATASOURCE_ONE,$DATASOURCE_TWOFor example:
$METRON_HOME/bin/start_parser_topology.sh -z $ZOOKEEPER_HOST:2181 -s bro,yaf
If you want to override parser parameters, you can add the parameter and its
value to the deployment command.
For a list of parser parameters, see Create a Parser for Your New Data Source by Using the CLI.For example:
$METRON_HOME/bin/start_parser_topology.sh -z $ZOOKEEPER_HOST:2181 -s bro,yaf -spoutNumTasks 2,3 -parserParallelism 2 -parserNumTasks 5This command will create a toplogy with the following parameters:
- Bro - spout number of tasks = 2
- YAF - spout number of tasks = 3
- YAF - paser parallelism = 2
- YAF - parser number of tasks = 5
- Use the Apache Storm UI to verify that the new topology is listed and that it has no errors.