Tuning Parameters for the Profiler Service
You can update the configuration settings to use minimal resources for the Profiler service.
Make sure the minimum resource requirements are as follows:
- RAM should be greater than or equal to 16 GB.
- CPU Cores should be greater than equal to 8.
Set the number of executors for Livy interactive sessions to 1.
- Log in to Ambari.
- From Ambari, select Dataplane Profiler and click Configs.
In the Advanced dpprofiler-livy-config section, set the following properties:
- Number Of Executors For Write Session =1
- Number Of Executors For Read Session=1
Set the number of executors for profiler jobs to 1.
- Log in to the Data Steward Studio app.
- Go to profiler configuration and click edit.
- Select a profiler. The Profile Configuration tab opens on the right.
- Update the details of the number of executors in the Advanced Options section. Set the number of executors to 1.
Update the YARN parameters as follows:
yarn.scheduler.capacity.maximum-am-resource-percentparameter on YARN > Scheduler (let this be x) such that
(x * total_memory_in_yarn) >= 8G
For example, for 16 GB it is advised to set x to 0.5.
All these resources must be allocated exclusively for profiler agent and profilers. It is advisable to have a separate queue.