Hortonworks Cybersecurity Platform
Also available as:
PDF
loading table of contents...

Add X-Pack Extension to Elasticsearch

You can add the X-Pack extension to Elasticsearch to enable secure connections for Elasticsearch.

Ensure that Elasticsearch and Kibana are installed. You must also choose the X-pack version that matches the version of Elasticsearch that you are running.
  1. Use the Storm UI to stop the random_access_indexing topolog
    1. From Topology Summary, click random_access_indexing
    2. Under Topology actions, click Deactivate.
  2. Install X-Pack on Elasticsearch and Kibana.
    See Installing X-Pack for information on installing X-Pack.
  3. After installing X-pack, navigate to the Elasticsearch node where Elasticsearch Master and the X-Pack were installed, then add a user name and password for Elasticsearch and Kibana to enable external connections from Metron components:
    For example, the following creates a user transport_client_user with the password changeme and superuser credentials:
    sudo /usr/share/elasticsearch/bin/x-pack/users useradd transport_client_user -p changeme -r superuser
  4. Create a file containing the password you created in Step 3 and upload it to HDFS.
    For example:
    echo changeme > /tmp/xpack-password
    sudo -u hdfs hdfs dfs -mkdir /apps/metron/elasticsearch/
    sudo -u hdfs hdfs dfs -put /tmp/xpack-password /apps/metron/elasticsearch/
    sudo -u hdfs hdfs dfs -chown metron:metron /apps/metron/elasticsearch/xpack-password
    
  5. Pull the most recent HCP configuration to the local file system by running the following on the node on which HCP is installed:
    $METRON_HOME/bin/zk_load_configs.sh -m PULL -o ${METRON_HOME}/config/zookeeper -z $ZOOKEEPER -f
  6. Set the X-Pack es.client.class by adding it to $METRON_HOME/config/zookeeper/global.json.
    For example, add the following to the global.json file:
    {
    ...
      "es.client.settings" : {
          "es.client.class" : "org.elasticsearch.xpack.client.PreBuiltXPackTransportClient",
          "es.xpack.username" : "transport_client_user",
          "es.xpack.password.file" : "/apps/metron/elasticsearch/xpack-password"
      }
      ...
    }
  7. OPTIONAL: To set SSL support for Elasticsearch X-pack, add the following properties to es.client.settings in the $METRON_HOME/config/zookeeper/global.json file:
    {
    ...
      "es.client.settings" : {
        "xpack.ssl.key": "/path/to/client.key",
        "xpack.ssl.certificate": "/path/to/client.crt",
        "xpack.ssl.certificate_authorities": "/path/to/ca.crt",
        "xpack.security.transport.ssl.enabled": "true" 
      }
      ...
    }
    Note
    Note
    Make sure you do not overwrite the existing es.client.settings properties.
    The client.key, client.crt, and ca.crt must reside on all Storm supervisor nodes as well as the REST application node.
    For more information about configuring Elasticsearch SSL for X-pack, see Java Client and Security.
  8. Add the X-Pack changes to ZooKeeper:
    $METRON_HOME/bin/zk_load_configs.sh -m PUSH -i METRON_HOME/config/zookeeper/ -z $ZOOKEEPER
  9. Create a custom X-Pack shaded and relocated jar file.
    Your jar file is specific to your licensing restrictions. However, you can use the following example for reference:
    <project xmlns="http://maven.apache.org/POM/4.0.0"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-xpack-shaded</artifactId>
        <name>elasticsearch-xpack-shaded</name>
        <packaging>jar</packaging>
        <version>5.6.2</version>
        <repositories>
            <repository>
                <id>elasticsearch-releases</id>
                <url>https://artifacts.elastic.co/maven</url>
                <releases>
                    <enabled>true</enabled>
                </releases>
                <snapshots>
                    <enabled>false</enabled>
                </snapshots>
            </repository>
        </repositories>
        <dependencies>
            <dependency>
                <groupId>org.elasticsearch.client</groupId>
                <artifactId>x-pack-transport</artifactId>
                <version>5.6.2</version>
                <exclusions>
                  <exclusion>
                    <groupId>com.fasterxml.jackson.dataformat</groupId>
                    <artifactId>jackson-dataformat-yaml</artifactId>
                  </exclusion>
                  <exclusion>
                    <groupId>com.fasterxml.jackson.dataformat</groupId>
                    <artifactId>jackson-dataformat-cbor</artifactId>
                  </exclusion>
                  <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-core</artifactId>
                  </exclusion>
                  <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                  </exclusion>
                  <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                  </exclusion>
                  <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                  </exclusion>
                  <exclusion> <!-- this is causing a weird build error if not excluded - Error creating shaded jar: null: IllegalArgumentException -->
                        <groupId>org.apache.logging.log4j</groupId>
                        <artifactId>log4j-api</artifactId>
                    </exclusion>
                </exclusions>
              </dependency>
        </dependencies>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-shade-plugin</artifactId>
                    <version>2.4.3</version>
                    <configuration>
                        <createDependencyReducedPom>true</createDependencyReducedPom>
                    </configuration>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>shade</goal>
                            </goals>
                            <configuration>
                              <filters>
                                <filter>
                                  <artifact>*:*</artifact>
                                  <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                  </excludes>
                                </filter>
                              </filters>
                              <relocations>
                                    <relocation>
                                        <pattern>org.apache.logging.log4j</pattern>
                                        <shadedPattern>org.apache.metron.logging.log4j</shadedPattern>
                                    </relocation>
                                </relocations>
                                <artifactSet>
                                    <excludes>
                                        <exclude>org.slf4j.impl*</exclude>
                                        <exclude>org.slf4j:slf4j-log4j*</exclude>
                                    </excludes>
                                </artifactSet>
                                <transformers>
                                    <transformer
                                      implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
                                         <resources>
                                            <resource>.yaml</resource>
                                            <resource>LICENSE.txt</resource>
                                            <resource>ASL2.0</resource>
                                            <resource>NOTICE.txt</resource>
                                          </resources>
                                    </transformer>
                                    <transformer
                                            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                    <transformer
                                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                        <mainClass></mainClass>
                                    </transformer>
                                </transformers>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </project>
  10. After you build the elasticsearch-xpack-shaded-5.6.2.jar file, you must make the file available to Storm when you submit the topology.
    Create a contrib directory for indexing and then put the elasticsearch-xpack-shaded-5.6.2.jar file in this directory:
    $METRON_HOME/indexing_contrib/elasticsearch-xpack-shaded-5.6.2.jar
  11. Use Ambari to restart the REST API.
  12. Use the Storm UI to restart the random_access_indexing topology.
    1. From Topology Summary, click random_access_indexing.
    2. Under Topology actions, click Start.