Creating a Cluster on AWS
Also available as:
PDF

Create a cluster on AWS

Use these steps to create a cluster with Cloudbreak.

If you experience problems during cluster creation, refer to Troubleshooting cluster creation.

Prerequisites

This prerequisite step is required for AWS GovCloud users only and is optional for other AWS users.

If you are planning to create clusters on AWS GovCloud, you must first prepare a custom image and register a custom image catalog with Cloudbreak. For instructions, refer to Custom images.

Steps

  1. Log in to the Cloudbreak web UI.
  2. Click the Create Cluster button and the Create Cluster wizard is displayed.

    By default, the Basic view is displayed. To view advanced options, click Advanced. To learn about advanced options, refer to Advanced cluster options.

  3. On the General Configuration page, specify the following general parameters for your cluster:
    Parameter Description
    Select Credential Choose a previously created credential.
    Cluster Name Enter a name for your cluster. The name must be between 5 and 40 characters, must start with a letter, and must only include lowercase letters, numbers, and hyphens.

    Region

    Select the AWS region in which you would like to launch your cluster. For information on available AWS regions, refer to AWS regions and endpoints in AWS documentation.
    Platform Version Choose the HDP or HDF version to use for this cluster. Blueprints available for this platform version will be populated under “Cluster Type” below. If you selected the HDF platform, refer to Creating HDF clusters for HDF cluster configuration tips.
    Cluster Type Choose one of the default cluster configurations, or, if you have defined your own cluster configuration via Ambari blueprint, you can choose it here. For more information on default and custom blueprints, refer to Custom blueprints.
    Flex Subscription This option will appear if you have configured your deployment for a flex subscription.
  4. On the Hardware and Storage page, for each host group provide the following information to define your cluster nodes and attached storage. To edit this section, click on the . When done editing, click on the to save the changes.
    Parameter Description
    Ambari Server You must select one node for Ambari Server by clicking the button. The “Instance Count” for that host group must be set to “1”. If you are using one of the default blueprints, this is set by default.
    Instance Type Select an instance type. For information about instance types on AWS refer to Amazon EC2 instance types in AWS documentation.
    Instance Count Enter the number of instances of a given type. Default is 1.
    Storage Type Select the volume type. The options are: (1) Magnetic (default), (2) General Purpose (SSD, (3) Throughput Optimized HDD. For more information about these options refer to Amazon EC2 Instance Store in AWS documentation.
    Encryption Under Encryption Key, you can select an existing encryption key. For more information, refer to EBS encryption.
    Attached Volumes Per Instance Enter the number of volumes attached per instance. Default is 1.
    Volume Size Enter the size in GB for each volume. Default is 100.
    Root Volume Size This option allows you to increase or decrease the root volume size. Default is 50 GB. This option is useful if your custom image requires more space than the default 50 GB.
    Use Spot Instances Check this option to use EC2 spot instances as your cluster nodes. Next, enter your bid price. The price that is pre-loaded in the form is the current on-demand price for your chosen EC2 instance type. For more information, refer to Use spot instances.
  5. On the Network and Availability page, provide the following to specify the networking resources that will be used for your cluster:
    Parameter Description
    Select Network Select the virtual network in which you would like your cluster to be provisioned. You can select an existing network or create a new network.
    Note
    Note

    The Shared Networks option is only available for Google Cloud.

    Select Subnet Select the subnet in which you would like your cluster to be provisioned. If you are using a new network, create a new subnet. If you are using an existing network, select an existing subnet.
    Subnet (CIDR) If you selected to create a new subnet, you must define a valid CIDR for the subnet. Default is 10.0.0.0/16.
    Note
    Note

    Cloudbreak uses public IP addresses when communicating with cluster nodes.

    On AWS, you can configure it to use private IPs instead. For instructions, refer to Configure communication via private IPs on AWS in the Troubleshooting documentation.

  6. On the Gateway Configuration page, you can access gateway configuration options.

    When creating a cluster, Cloudbreak installs and configures a gateway (powered by Apache Knox) to protect access to the cluster resources. By default, the gateway is enabled for Ambari; You can optionally enable it for other cluster services.

    For more information, refer to Gateway configuration documentation.

  7. On the Network Security Groups page, define security groups for each host group. You can either create new security groups and define their rules or reuse existing security groups:
    Note
    Note

    Existing security groups are only available when an existing VPC is selected.

    Option Description
    New Security Group (Default) Creates a new security group with the rules that you defined:
    • A set of default rules is provided. You should review and adjust these default rules. If you do not make any modifications, default rules will be applied.
    • You may open ports by defining the CIDR, entering port range, selecting protocol and clicking +.
    • You may delete default or previously added rules using the delete icon.
    • If you don't want to use security group, remove the default rules.
    Existing Security Groups Allows you to select one or more existing security groups that exist in the selected region and network. To use an existing security group, select it from the dropdown and then click "Attach". Repeat these steps if you would like to use multiple security groups. This selection is disabled if no existing security groups are available in your chosen region and network.

    The default experience of creating network resources such as network, subnet and security group automatically is provided for convenience. We strongly recommend you review these options and for production cluster deployments leverage your existing network resources that you have defined and validated to meet your enterprise requirements. For more information, refer to Restrict inbound access to clusters.

  8. On the Security page, provide the following parameters:
    Parameter Description
    Cluster User You can log in to the Ambari web UI using this username. By default, this is set to admin.
    Password You can log in to the Ambari web UI using this password.
    Confirm Password Confirm the password.
    New SSH public key Check this option to specify a new public key and then enter the public key. You will use the matching private key to access your cluster nodes via SSH.
    Existing SSH public key Select an existing public key. You will use the matching private key to access your cluster nodes via SSH. This is a default option as long as an existing SSH public key is available.
  9. Click on Create Cluster to create a cluster.
  10. You will be redirected to the Cloudbreak dashboard, and a new tile representing your cluster will appear at the top of the page.