2.3. Common HDFS Commands

This section provides common HDFS commands to troubleshoot HDP deployment on Windows platform. An exhaustive list of the commands is available at here.

  • Get Hadoop version:

    Execute the following command on your cluster host machine:

    hadoop version
  • Check block information: This command provides a directory listing and displays which node contains the block.  Use this command to determine if a block is under replicated.

    Execute the following command on your HDFS cluster host machine:

    hadoop fsck / -blocks -locations -files
    

    You should see output similar to the following:

     FSCK started by hdfs from /10.0.3.15 for path / at Tue Feb 12 04:06:18 PST 2013 
       / <dir> 
       /apps <dir> 
       /apps/hbase <dir> 
       /apps/hbase/data <dir> 
       /apps/hbase/data/-ROOT- <dir> 
       /apps/hbase/data/-ROOT-/.tableinfo.0000000001 727 bytes, 1 block(s):  
        Under replicated blk_-3081593132029220269_1008. 
       Target Replicas is 3 but found 1 replica(s). 0. blk_-3081593132029220269_1008 
        len=727 repl=1 [10.0.3.15:50010]   
        /apps/hbase/data/-ROOT-/.tmp <dir> 
        /apps/hbase/data/-ROOT-/70236052 <dir> 
        /apps/hbase/data/-ROOT-/70236052/.oldlogs <dir> 
        /apps/hbase/data/-ROOT-/70236052/.oldlogs/hlog.1360352391409 421 bytes, 1 block(s):  Under
            replicated blk_709473237440669041_1006. 
          Target Replicas is 3 but found 1
          replica(s). 0. blk_709473237440669041_1006 len=421 repl=1 [10.0.3.15:50010] ... 

  • HDFS report: Use this command to receive HDFS status.

    Execute the following command as an HDFS user:

    hadoop dfsadmin -report

    You should see output similar to the following:

    -bash-4.1$ hadoop dfsadmin -report
    Safe mode is ON
    Configured Capacity: 11543003135 (10.75 GB)
    Present Capacity: 4097507328 (3.82 GB)
    DFS Remaining: 3914780672 (3.65 GB)
    DFS Used: 182726656 (174.26 MB)
    DFS Used%: 4.46%
    Under replicated blocks: 289
    Blocks with corrupt replicas: 0
    Missing blocks: 0
    
    -------------------------------------------------
    Datanodes available: 1 (1 total, 0 dead)
    
    Name: 10.0.3.15:50010
    Decommission Status : Normal
    Configured Capacity: 11543003135 (10.75 GB)
    DFS Used: 182726656 (174.26 MB)
    Non DFS Used: 7445495807 (6.93 GB)
    DFS Remaining: 3914780672(3.65 GB)
    DFS Used%: 1.58%
    DFS Remaining%: 33.91%
    Last contact: Sat Feb 09 13:34:54 PST 2013
    

  • Safemode: Safemode is a state where no changes can be made to the blocks.  HDFS cluster is in safemode state during start up because the cluster needs to validate all the blocks and their locations. Once validated, the safemode is then disabled.

    The options for safemode command are:

    hadoop dfsadmin -safemode [enter | leave | get]

    To enter the safemode, execute the following command on your NameNode host machine:

    hadoop dfsadmin -safemode enter


loading table of contents...