14. Validating Your Data

Verify that your data is intact by comparing the HDFS data directory tree with the HDP 1.3 or HDP 2.0 tree.

  1. Run the following command to open a command prompt with the HDFS user in the Hadoop Home directory:

    runas /user:hadoop "cmd /K cd %HDFS_HOME%\bin"
  2. Run an lsr report on your upgraded system. Execute the following command from the Hadoop command line:

    hadoop fs -lsr / > dfs-new-lsr-1.log    
  3. Compare the directory listing to the older HDP directories. All old directories, files and timestamps should match. There will be some new entries in the HDP directory listing:

    • /apps/hbase is only in HDP and is used by HBase (new when upgrading from 1.3 to 2.1)

    • /mapred/system/jobtracker will have a new timestamp

  4. Run a fsck report on your upgraded system. Execute the following command from the Hadoop Command Line:

    hdfs fsck / -blocks -locations -files fsck-new-report-1.log
  5. Compare this fsck report to the prior to upgrade report to check the validity of your current HDFS data.


loading table of contents...