Data Steward Studio Installation
Also available as:
PDF

Tuning Parameters for the Profiler Service

You can update the configuration settings to use minimal resources for the Profiler service.

Make sure the minimum resource requirements are as follows:

  • RAM should be greater than or equal to 16 GB.
  • CPU Cores should be greater than equal to 8.
  1. Set the number of executors for Livy interactive sessions to 1.
    1. Log in to Ambari.
    2. From Ambari, select Dataplane Profiler and click Configs.
    3. In the Advanced dpprofiler-livy-config section, set the following properties:
      • Number Of Executors For Write Session =1
      • Number Of Executors For Read Session=1
  2. Set the number of executors for profiler jobs to 1.
    1. Log in to the Data Steward Studio app.
    2. Go to profiler configuration and click edit.
    3. Select a profiler. The Profile Configuration tab opens on the right.
    4. Update the details of the number of executors in the Advanced Options section. Set the number of executors to 1.
  3. Update the YARN parameters as follows:

    Set the yarn.scheduler.capacity.maximum-am-resource-percent parameter on YARN > Scheduler (let this be x) such that

    (x * total_memory_in_yarn) >= 8G

    For example, for 16 GB it is advised to set x to 0.5.

    All these resources must be allocated exclusively for profiler agent and profilers. It is advisable to have a separate queue.