For each cluster you can optionally enable auto scaling, which allows you to define one or more policies to automatically increase or decrease the capacity of the cluster when certain pre-defined conditions are met.
When creating an auto scaling policy, you define:
- The scaling event that will trigger the scaling policy.
- The scaling policy that will add or remove a set number of worker or compute nodes based on a scaling metric definition.
|Scaling Event||Scaling Policy|
|If CPU usage is more than 70% for 5 minutes (averaged across all cluster nodes).||Add 3 compute nodes|
|If HDFS usage is less than 10% for 15 minutes.||Remove 5 worker nodes|
|If Memory usage is more than 50% for 10 minutes (averaged across all cluster nodes).||Add 3 compute nodes|
In addition, you can configure cluster auto scaling settings that are valid for all auto scaling policies attached to a given cluster. These settings include:
- Enable or disable auto scaling
- Determine the length of the cooldown period that occurs after an auto scaling event is executed
- Define minimum and maximum cluster size
By default, the cooldown period lasts 30 minutes and the cluster size is between 3 and 100 nodes.
Enabling Cluster Auto Scaling
- When creating a cluster by using the AUTO SCALING section, or
- When the cluster is already running, from the cluster details page, either by using the options available in the CLUSTER ACTIONS > Auto Scaling menu or by using the options in the AUTO SCALING tab:
Viewing and Managing Cluster Auto Scaling
For any running cluster, you can view or manage auto scaling from the AUTO SCALING tab on the cluster details page:
- Click to enable or disable auto scaling
- Click to add a new auto scaling policy
- Click to remove an existing auto scaling policy
- Click to configure the auto scaling settings
Defining An Auto Scaling Policy
You can add one or more auto scaling policies for your cluster either when creating the cluster or when the cluster is already running.
In order to define an auto scaling policy, you need to provide the following information:
|Policy Name||A unique name for your policy (min 5, max 20, must start with a letter, accepted chars: a-z, 0-9, -)|
|Scaling Definition Metric||The scaling metric definition to watch|
|Metric Value||A value from 1 to 100 (percent); When the metric defined above reaches this value, the scaling policy will be executed|
|Duration||The amount of time (in minutes) the metric value needs to be more/less than defined above|
|Action||Select Add or Remove|
|Nodes||The number of nodes to add or remove|
|Group||The node type (worker or compute) to add or remove|
Scaling Metric Definitions
The following scaling metric definitions (or "scaling definitions" for short) can be used to define scaling events:
|CPU usage||Measures the CPU usage average across all nodes in the cluster.|
|Memory usage||Measures the Memory usage average across all nodes in the cluster.|
|HDFS usage||Measure the HDFS usage for the cluster.|
Configuring Cluster Auto Scaling Settings
You can configure auto scaling settings for your cluster either when creating the cluster or when the cluster is already running.
In order to configure the auto scaling settings for your cluster, you need to provide the following information:
|Cooldown Time||After an auto scaling event occurs, the amount of time to wait before enforcing another scaling policy.||30 minutes|
|Cluster Size Limit||The minimum and maximum size allowed for the cluster (worker + compute nodes). Auto scaling policies cannot scale the cluster above or below this size.||Min 3 nodes, max 100 nodes|
Cluster Auto Scaling via CLI
You can also enable autoscaling, add autoscaling policies, and configure cluster auto scaling settings using the CLI. For more information, refer to Configuring Auto Scaling for Your Cluster.