Tutorial: Getting Started with HDCloud CLI

This tutorial will help you get started with the HDCloud CLI.

Overview

In this tutorial, we will work with the CLI available in the cloud controller UI. We will go through the following steps:

Let's get started!

Prerequisites:

Before starting this tutorial, your cloud controller needs to be running and you must to be logged in to the cloud controller UI. To set up the cloud controller, refer to this tutorial.

Download the CLI

In order to use HDCloud CLI, you need to download it:

  1. Log in to the cloud controller UI.

  2. From the menu, select DOWNLOAD CLI and select your platform (Linux, Mac OS X, or Windows). I’m using Mac OS X.

  3. Clicking on the platform name should download a tgz file. On my Mac, it gets downloaded to the Downloads directory.

  4. Extract the file. For example, on a Mac you can do this by running:

    tar xvf hdc-cli_Darwin_x86_64.tgz

  5. From the directory to which you extracted the file, run ./hdc to verify that the hdc-cli works on your system.

  6. Optionally, you can add hdc to your PATH.

Configure the CLI

Once you have installed the hdc-cli, you can configure it for your cloud controller.

  1. To do that, use the hdc configure command and include the following parameters:

    --server cloud controller URL
    --username cloud controller user name (email address)
    --password cloud controller password

    You can obtain the cloud controller URL from the browser address bar (if you have your cloud controller UI open) or from the CloudFormation console's Outputs:

    The password configuration is optional. If you do not provide the password, no password is stored in the CLI configuration file. Therefore, you will need to provide the password with each command you execute or via an environment variable.

    For example:

    ./hdc configure --server https://ec2-34-208-50-241.us-west-2.compute.amazonaws.com --username dbialek@hortonworks.com --password My1Secret2Password3

  2. This will create a ~/.hdc/config file. The file will include the configuration information that you specified.

    For example:

    username: dbialek@hortonworks.com
    password: My1Secret2Password3
    server: https://ec2-34-208-50-241.us-west-2.compute.amazonaws.com
    

If you are working with multiple cloud controllers, you can specify the configuration information when executing hdc-cli commands.

Create a CLI JSON File

To create a cluster, we need to obtain and edit the CLI JSON skeleton, save it in a JSON file and then create a cluster based on the specifications provided in the JSON file:

  1. There are a few ways to obtain the CLI JSON skeleton generate the skeleton.

    Method 1: From the HDCloud CLI, you can do this by running:

    ./hdc create-cluster generate-cli-skeleton

    This will generate a cluster template in JSON format.

    Method 2: Alternatively, you can obtain the CLI JSON skeleton from the cloud controller UI.

    For example, you can provide the parameters in the create cluster form, click CREATE CLUSTER, and then click SHOW CLI JSON.

    Method 3: Or, you can obtain the JSON template from saved templates in the cloud controller UI:

    Click on SHOW CLI JSON to view the template and then click on the copy icon to copy the content:

  2. Copy and paste the template into a text editor and save it as a JSON file.

  3. In the JSON file, specify the following mandatory parameters:

    "ClusterName”
    "SSHKeyName"
    "RemoteAccess"
    "ClusterAndAmbariUser"
    "ClusterAndAmbariPassword"

    For example, my CLI JSON skeleton file is called "cluster_template.JSON", and it looks like this:

    {
    "ClusterName": "test1234",
    "HDPVersion": "2.6",
    "ClusterType": "EDW-ETL: Apache Hive 1.2.1, Apache Spark 2.1",
    "Master": {
        "InstanceType": "m4.4xlarge",
        "VolumeType": "gp2",
        "VolumeSize": 32,
        "VolumeCount": 1,
        "Encrypted": false
    },
    "Worker": {
        "InstanceType": "m3.xlarge",
        "VolumeType": "ephemeral",
        "VolumeSize": 40,
        "VolumeCount": 2,
        "Encrypted": false,
        "InstanceCount": 3,
        "RecoveryMode": "AUTO"
    },
    "Compute": {
        "InstanceType": "m3.xlarge",
        "VolumeType": "ephemeral",
        "VolumeSize": 40,
        "VolumeCount": 2,
        "Encrypted": false,
        "InstanceCount": 0,
        "RecoveryMode": "AUTO"
    },
    "SSHKeyName": "dominika-kp",
    "RemoteAccess": "0.0.0.0/0",
    "WebAccess": true,
    "HiveJDBCAccess": true,
    "ClusterComponentAccess": true,
    "ClusterAndAmbariUser": "admin",
    "ClusterAndAmbariPassword": "My1Secure2Password3",
    "Tags": {},
    "Autoscaling": {
        "Configurations": {
            "CooldownTime": 30,
            "ClusterMinSize": 3,
            "ClusterMaxSize": 100
        }
    },
    "InstanceRole": "CREATE"
    }
    

    To get an example of a complete CLI JSON skeleton, run

    ./hdc create-cluster generate-cli-skeleton --help

    You can also consult CLI documentation.

  4. Optionally, you can also change the “ClusterType”. To check for valid cluster types, run:

    ./hdc list-cluster-types

Create a Cluster

  1. Once you provided the parameters, you can create a cluster. For example, since my CLI JSON skeleton file is called "cluster_template.json", I ran:

    ./hdc create-cluster -cli-input-json cluster_template.json

  2. If everything goes well, you should see a new cluster being created in your cloud controller UI:

    While your cluster is being created, you should also be able to see it on the List of available clusters and check its status and health:

    ./hdc list-clusters --output table

    And you should be able to get CLI JSON for this cluster:

    ./hdc describe-cluster --cluster-name test1234

Run More Commands

Here is a list of other useful commands that you can try once your cluster has been created:

  1. To see details of the instances in the newly created cluster:

    .hdc describe-cluster instances --cluster-name test1234

  2. To add one compute node to your cluster:

    ./hdc resize-cluster --cluster-name test1234 --node-type compute --scaling-adjustment 1

  3. To remove one worker node from your cluster:

    ./hdc resize-cluster --cluster-name test1234 --node-type worker --scaling-adjustment -1

  4. Once you are done, you can terminate your cluster using the following command:

    ./hdc terminate-cluster --cluster-name test1234

For more commands, refer to CLI documentation.