IOP to HDP Migration
Also available as:
PDF

Chapter 4. Back Up Configurations

Next, make backups of the following configurations:

Backup WebHcat Configuration

WebHcat configuration might be erased when HDP rpm package is uninstalled. It is recommended to backup WebHcat configuration on WebHcat node for rollback.

mkdir -p /tmp/backup
tar cfvz /tmp/backup/webhcat.tar.gz /etc/hive-webhcat

Back up Hcatalog Configuration

Hcatalog configuration might be erased when HDP rpm package is uninstalled. It is recommended to backup Hcatalog configuration on Hcatlog node for rollback.

tar cfvz /tmp/backup/hcatalog.tar.gz /etc/hive-hcatalog

Back up Spark Configuration

Spark configuration will be erased when HDP rpm package is uninstalled. It is recommended to backup Spark configuration on all nodes for rollback.

tar cfvz /tmp/backup/spark.tar.gz /etc/spark

Back up Solr

This topic describes the steps necessary to back up your Solr index, configurations, data, and core instance.

Preparing Solr for Backups

1. For each replica, optimize the index.

2. Verify that for each replica, the value of num_docs is equal to the value of total_docs.

3. Verify that all Collections are healthy.

4. Verify that all Shards have live leaders.

5. Verify that all Replicas of a collection are healthy.

6. Record the number of documents in each replica.

7. Stop Solr and run CheckHDFSIndex for each replica to verify index health.

8. Start Solr.

Back up Jars

  1. Backup all custom jars added to Solr’s classpath.

  2. The jars in the classpath can be seen on the Solr UI unde Java properties.

  3. For each replica, check status of backup.

Back up Index

  1. Create a directory owned by Solr to store the backups.

  2. For each replica, create a backup.

  3. Do not issue multiple backup commands on the same replica!

Verify Backups

  1. For each backup, run CheckIndex.

  2. Verify index integrity.

Example Code:

#CheckIndex
 su - solr
 cd /usr/iop/4.2.0.0/solr
 java -cp "$(hadoop classpath):
 ./server/solr-webapp/webapp/WEB-INF/lib/*:./server/lib/ext/*:/usr/iop/current/hadoop-client/lib/servlet-api-2.5.jar"
 -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex
 /tmp/backups/snapshot.c1_shard1_replica1/

#CheckHDFSIndex
 su - solr
 cd /usr/iop/4.2.0.0/solr
 java -cp "$(hadoop classpath):
 ./server/solr-webapp/webapp/WEB-INF/lib/*:./server/lib/ext/*:/usr/iop/current/hadoop-client/lib/servlet-api-2.5.jar"
 -ea:org.apache.lucene...
 org.apache.solr.index.hdfs.CheckHdfsIndex
 hdfs://node1.ibm.com:8020/apps/solr/data/c1/core_node1/data/index

#Backup
 su - solr

 curl
'http://localhost:8983/solr/c1_shard1_replica1/replication?command=backup&name=c1_shard1_replica1&location=/tmp/backups'

#Backup Status
 curl
 'http://localhost:8983/solr/c1_shard1_replica1/replication?command=details' |
 xmllint --format 

Backup Configuration:

  1. Download migration_helper.py from

    https://github.com/hortonworks/iop-solr-stack/releases/download/iop-solr-1.0.0.0-migration/migration_helper.py.

    or download the Solr mpack from

    http://s3.amazonaws.com/dev.hortonworks.com/IOPSOLR/centos7/1.x/BUILDS/1.0.0.0-24/tars/iopsolr/iop-solr-mpack-6.3.0.1.0.0.0-24.tar.gz

    (replace centos 7 with centos6 in the path if the machine is running centos6 OS). Untar that tarball, and find the migration_helper.py in the \utilities folder.

  2. Backup configurations

    1. Create a backup folder.

      For example:

      mkdir -p /tmp/solr-backup
    2. Run migration_helper.py with --action backup flag:

      /usr/bin/python migration_helper.py --action backup --backup-location /tmp/solr-backup
      --cluster <cluster-name> -H <ambari-server-host> --username
      <ambari-admin-username> --password <ambari-admin-password> --port <ambari-server-port>
    [Note]Note

    This operation will generate SOLR-backup-*.json files.

    Ambari credentials/host/port are used here.

    Use -s if ambari-server uses https.

  3. Backup solr core (optional, but recommended):

    "curl http://<solr-server>:8983/solr/<core_name>/replication?command=backup&location=<backup-location>

    Replace core name and backup location with actual ones.

    [Note]Note

    Do not forget kinit and --negotaiate -u: flags in the curl command if Solr uses kerberos.

    If you are migrating from IOP 4.2.0, see https://cwiki.apache.org/confluence/display/solr/Making+and+Restoring+Backups