Planning for the HDP Cluster
Also available as:

File System Partitioning Recommendations

You can implement various partitioning recommendations for the file system when deploying your Hadoop cluster.

Setting Up File System Partitions

Use the following as a base configuration for all nodes in your cluster:

  • Root partition: OS and core program files

  • Swap: Size 2X system memory

Partitioning Recommendations for Slave Nodes

  • Hadoop Slave node partitions: Hadoop should have its own partitions for Hadoop files and logs. Drives should be partitioned using XFS, or ext4 in that order of preference. Do not use LVM; it adds latency and causes a bottleneck.

  • On slave nodes only, all Hadoop partitions should be mounted individually from drives as "/grid/[0-n]".

  • Hadoop Slave Node Partitioning Configuration Example:

    • /root - 20GB (ample room for existing files, future log file growth, and OS upgrades)

    • /grid/0/ - [full disk GB] first partition for Hadoop to use for local storage

    • /grid/1/ - second partition for Hadoop to use

    • /grid/2/ - ...

Redundancy (RAID) Recommendations

  • Master nodes -- Configured for reliability (RAID 10, dual Ethernet cards, dual power supplies, etc.)

  • Slave nodes -- RAID is not necessary, as failure on these nodes is managed automatically by the cluster. All data is stored across at least three different hosts, and therefore redundancy is built-in. Slave nodes should be built for speed and low cost.

Further Reading

The following additional documentation may be useful: