YARN Resource Management
Also available as:
PDF
loading table of contents...

Default Queue Mapping based on User or Group

Administrators can define a default mapping policy to specify that applications submitted by users are automatically submitted to queues. With a default mapping policy, users are not required to specify the queue name when submitting their applications. The default mapping policy can be configured to be overridden if the queue name is specified for the submitted application.

Queue mapping is defined using a comma-separated list of mapping assignments. The order of the mapping assignments list is important -- in cases where mulitple mapping assignments are used, the Capacity Scheduler processes the mapping assignments in left-to-right order to determine which mapping assignment to use first.

The Queue mapping assignment is defined using the yarn.scheduler.capacity.queue-mappings property in the capacity-scheduler.xml file. Queue mapping assignments can be defined for a user (using "u") or for a group of users (using "g"). Each mapping assignment type is described in the following sections.

Configuring Queue Mapping for Users and Groups to Specific Queues

To specify that all applications submitted by a specific user are submitted to a specific queue, use the following mapping assignment:

u:user1:queueA

This defines a mapping assignment for applications submitted by the "user1" user to be submitted to queue "queueA" by default.

To specify that all applications submitted by a specific group of users are submitted to a specific queue, use the following mapping assignment:

g:group1:queueB

This defines a mapping assignment for applications submitted by any user in the group "group1" to be submitted to queue "queueB" by default.

The Queue Mapping definition can consist of multiple assignments, in order of priority.

Consider the following example:

<property>
  <name>yarn.scheduler.capacity.queue-mappings</name>
  <value>u:maria:engineering,g:webadmins:weblog</value>
</property>

In this example there are two queue mapping assignments. The u:maria:engineering mapping will be respected first, which means all applications submitted by the user "maria" will be submitted to the "engineering" queue . The g:webadmins:weblog mapping will be processed after the first mapping -- thus, even if user "maria" belongs to the "webadmins" group, applications submitted by "maria" will still be submitted to the "engineering" queue.

Configuring Queue Mapping for Users and Groups to Queues with the Same Name

To specify that all applications are submitted to the queue with the same name as a group, use this mapping assignment:

u:%user:%primary_group

Consider the following example configuration. On this cluster, there are two groups: "marketing" and "engineering". Each group has the following users:

In "marketing", there are 3 users: "angela", "rahul", and "dmitry".

In "engineering", there are 2 users: "maria" and "greg".

<property>
  <name>yarn.scheduler.capacity.queue-mappings</name>
  <value>u:%user:%primary_group</value>
</property>

With this queue mapping, any application submitted by members of the "marketing" group -- "angela", "rahul", or "dmitry" -- will be submitted to the "marketing" queue. Any application submitted by members of the "engineering" group -- "maria" or "greg" -- will be submitted to the "engineering" queue.

To specify that all applications are submitted to the queue with the same name as a user, use this mapping assignment:

u:%user:%user

This requires that queues are set up with the same name as the users. With this queue mapping, applications submitted by user "greg" will be submitted to the queue "greg".

Enabling Override of Default Queue Mappings

If configured, you can override default queue mappings and submit applications that are specified for queues, other than those defined in the default queue mappings. Override default queue mapping is disabled (set to false) by default.

<property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a queue mapping is present and override is set to true, it will override the queue value specified
      by the user. This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false - user can specify to a non-default queue.
    </description>
</property>

To enable queue mapping override, set the property to true in the capacity-scheduler.xml file.

Consider the following example in the case where queue mapping override has been enabled:

<property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value>u:maria:engineering,g:webadmins:weblog</value>
</property> 

If user "maria" explicitly submits an application to the "marketing" queue, the default queue assignment of "engineering" is overridden, and the application is submitted to the "marketing" queue.