Getting Started with Streaming Analytics
Also available as:
PDF
loading table of contents...

Registering a Custom Source in SAM for AWS Kinesis

To register any custom source in SAM, there are three artifacts you need:

  1. Artifact 1: Code for the custom source using the underlying streaming engine. Since SAM today supports Storm as the Streaming engine, you can refer to the following artifacts for the custom source:

  2. Artifact 2: Code for mapping the SAM configs to the custom source/spout. Refer to the following artifacts for this mapping code:

  3. Artifact 3: Flux mapping file to map the SAM config to the Kinesis Spout. Refer to the following artifacts

More Details on implementing a custom source and registering with SAM can be found here: https://github.com/hortonworks/streamline/tree/master/examples/sources

To register the custom Kinesis Source in SAM using the above three artifacts, perform the following steps:

  1. Download the Sam-Custom-Extensions.zip to the host where SAM is installed (if you haven’t done it in a past step)

  2. Unzip the contents. We will call the unzipped folder $SAM_EXTENSIONS

  3. Switch to user streamline:

    sudo su streamline
  4. Install Artifact 1 (the custom source code) on host’s local maven repo

    cd $SAM_EXTENSIONS/custom-source/kinesis/
    mvn install:install-file -Dfile=storm-kinesis-1.1.0.5.jar \
    -DgroupId=org.apache.storm \
    -DartifactId=storm-kinesis \
    -Dversion=1.1.0.5 \
    -Dpackaging=jar
    
  5. Register the custom source via SAM REST call. Replace SAM_HOST and SAM_PORT.

    curl -sS -X POST -i -F \
    topologyComponentBundle=@config/kinesis-source-topology-component.json -F \
    bundleJar=@sam-custom-source-kinesis.jar \
    http://SAM_HOST:SAM_PORT/api/v1/catalog/streams/componentbundles/SOURCE
  6. If the registration was successful, you should see a message like the following by the REST response:

    HTTP/1.1 201 Created
    Date: Wed, 03 Jan 2018 20:26:22 GMT
    Content-Type: application/json
    Content-Length: 4569
    
  7. On the SAM Application Canvas Palette, you should now see KINESIS source.

  8. Dragging the kinesis source onto the canvas and double clicking it, you should see the following kinesis dialog. The dialog properties comes from the topologyComponentBundle flux config you used to register the custom source.