IOP to HDP Migration
Also available as:
PDF

Solr Migration

After HDP 2.6 Upgrade completes successfully, there are 2 options to consider:

  1. IOP SOLR is being actively used by the user outside of the use by Ranger and thereby SOLR has custom collections that need to be migrated. This requires installation of the IOP SOLR management pack after the upgrade.

    [Note]Note

    If Solr was used for Ranger/Atlas as well, you can go ahead with the mpack install or use option 2.

  2. There are no custom collections and is it ok to lose Ranger audit data (if Solr was used with Ranger before the upgrade). In that case, installing the IOP SOLR management pack is not required. After upgrading to HDP 2.6.2, the only required step is to use the Add Service wizard to install Ambari Infra service.

If Option 1 above is true, install the IOP SOLR using management pack using the following instructions:

  1. Download the IOP Solr Management Pack.

    http://s3.amazonaws.com/dev.hortonworks.com/IOPSOLR/centos7/1.x/BUILDS/1.0.0.0-24/tars/iopsolr/iop-solr-mpack-6.3.0.1.0.0.0-24.tar.gz
  2. Stop ambari server.

    ambari-server stop
  3. Install the IOP Solr Management Pack.

    ambari-server install-mpack --mpack=/PATH/TO/iop-solr-mpack-VERSION.tar.gz

    Replace the path with the absolute path to the IOP Solr Management Pack.

  4. Restart ambari-server.

    ambari-server restart
  5. If you removed Solr before and created a backup (see: Backup SOLR), use migration_helper.py script to apply the old configurations. Before running the migration helper script, it’s required to create a json file for the secrets (usually keystore/truststore passwords). (as we cannot get those values through the rest api), you can find a sample in the utilities directory (after you untar the mpack tar.gz file),

    If you did not use SSL for Solr before the upgrade, you can use the default values, also the same is true for the other ranger properties if you did not used ranger before. Then you can run the script:

    /usr/bin/python migration_helper.py --action configure
    --backup-location /tmp/solr-backup --ranger-solr-secrets=secret_defaults.json
    --cluster <cluster-name> -H <ambari-server-host> --username
    <ambari-admin-username> --password <ambari-admin-password> --port
    <ambari-server-port> 
    # use -s if ambari uses ssl
  6. If the last command finished successfully, you can install IOP Solr Service with the migration_helper.py:

    /usr/bin/python migration_helper.py --action install --cluster <cluster-name> -H 
    <ambari-server-host> --username <ambari-admin-username> --password <ambari-admin-password> 
    --port <ambari-server-port> 
    # use -s if ambari uses ssl
  7. You have 2 options here:

    • You can start your Solr service using Ambari Web UI (you can check the configuration on the UI before starting the Solr instances)

    • You can use the migration_helper.py script to start the Solr service; just make sure the install has already finished:

      /usr/bin/python migration_helper.py --action start --cluster <cluster-name> -H 
      <ambari-server-host> --username <ambari-admin-username> --password <ambari-admin-password> 
      --port <ambari-server-port> 
      # use -s if ambari uses ssl
    [Note]Note
    • if something goes wrong, you can remove the service with --action stop and --action remove commands (use the same way as --action install or --action start)

    • The --action configure command will generate json files in the backup folder for every config type (with that name pattern: config-SOLR-<config_type>.json), those will be used as inputs to create the new configs and the host components.

  8. Post Upgrade steps to enable audits to solr in Ranger using IOP-mpack:

    • Change the following configurations from Ranger Audit Tab in Audit to Solr section:

      1. Enable Audit to Solr (xasecure.audit.destination.solr/ranger-env)

      2. Enable SolrCloud (is_solrCloud_enabled/ranger/env)

      3. Enable External SolrCloud(is_external_solrCloud_enabled/ranger-env)

      4. For secure cluster only Enable External SolrCloud kerberos (is_external_solrCloud_kerberos/ranger-env)

    • When External SolrCloud is enabled, Ambari will recommend to use /ranger_audits znode, change it to the znode setup by IOP-SOLR mpack (default znode is /solr) in ranger.audit.solr.zookeepers config property.

  9. Ranger Solr plug-in migration steps:

    Verify that the solr-solrj-5.5.x jar is available in the follwing path:

    /usr/iopsolr/current/iop-solr/server/solr-webapp/webapp/WEB-INF/lib/ranger-solr-plugin-impl/.

    If not, copy it from /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/lib/.

  10. Different migration steps are appropriate for each of the two, following existing setups. Perform the step(s) appropriate for your environment.

    Existing setupMigration stepsNotes

    IOP 4.2.0

    Ranger audits in DB

    Add Ambari Infra for storing audits.

    For Secure cluster:

    After upgrade to HDP-2.6.2, you must follow the steps for Ranger with Kerberos, in Post-upgrade Tasks for Ranger with Kerberos.

    IOP 4.2.5

    Ranger audits in Ambari Infra

    You must upgrade Ambari Infra. No further actions requiredAfter stack migration.