You can manage
resources for the applications running on your cluster by
resources through scheduling, limiting CPU usage by configuring cgroups, partitioning the
using node labels, and launching applications on Docker
Using Scheduling to Allocate Resources You can allocate CPU, GPU, and memory among users and groups in a Hadoop cluster. You can use scheduling to allocate the best possible nodes for application containers. GPU support for Docker You can use GPUs in big data applications such as machine learning, data analytics, and genome sequencing. Docker containerization makes it easier for you to package and distribute applications. You can enable GPU support when using YARN on Docker containers. Limit CPU Usage with Cgroups You can use cgroups to limit CPU usage in a Hadoop Cluster. Run Docker Containers on YARN You can configure YARN to run Docker containers. Using Docker Containers on YARN for Spark Jobs Apache Spark applications might have complex software dependencies which introduce package isolation challenges, especially in situations when you have to install multiple versions of the dependencies on cluster hosts where Spark executors run. Docker containers on YARN address such package isolation challenges by enabling you to install and manage the software dependencies as separate images on the containers. Partition a Cluster Using Node Labels You can use Node labels to partition a cluster into sub-clusters so that jobs run on nodes with specific characteristics.