yarn.nodemanager.resource.memory-mb 

Description: The total amount of physical memory (in MB) that YARN will allow containers to use on a NodeManager. 

Value: Set this to a value less than the total physical RAM on the server (e.g., 75-80% of total RAM) to leave memory for the operating system, Hadoop daemons (DataNode, NodeManager itself), and other system processes. 

yarn.nodemanager.resource.cpu-vcores 

Description: The total number of virtual CPU cores that YARN will allow containers to use on a NodeManager. 

Value: Set this equal to the total number of physical CPU cores on the server. Hyper-threading cores are generally counted as 1 physical core for this setting to avoid over-subscription. 

yarn.scheduler.minimum-allocation-mb 

Description: The minimum memory allocation request for any container (in MB). All container requests will be rounded up to this value. 

Value: 1024 (1GB) or 2048 (2GB) are common. Setting this too low can lead to an excessive number of small containers, increasing scheduling overhead. 

yarn.scheduler.maximum-allocation-mb 

Description: The maximum memory allocation request for any container (in MB). No container can request more memory than this. 

Value: Should not exceed yarn.nodemanager.resource.memory-mb. Set slightly lower to account for NodeManager overhead. 

yarn.scheduler.minimum-allocation-vcores 

Description: The minimum virtual CPU cores allocation request for any container. All container requests will be rounded up to this value. 

Value: 1 

yarn.scheduler.maximum-allocation-vcores 

Description: The maximum virtual CPU cores allocation request for any container. 

Value: Should not exceed yarn.nodemanager.resource.cpu-vcores. 

yarn.resourcemanager.scheduler.class 

Description: Specifies the YARN scheduler to use. 

Value: 

yarn.scheduler.capacity.root.queues 

Description: Defines the top-level queues under the root. 

Value: Comma-separated list of queue names (e.g., default,dev,prod). 

yarn.scheduler.capacity.<queue-path>.capacity: 

Description: The percentage of cluster resources guaranteed to a specific queue. The sum of direct children under a parent must be 100%. 

Value: Percentage (e.g., 50). 

yarn.scheduler.capacity.<queue-path>.maximum-capacity 

Description: The maximum percentage of cluster resources a queue can use, allowing it to "burst" above its guaranteed capacity if other queues are idle. 

Value: Percentage (e.g., 100). Set to 100 for dynamic resource sharing. 

yarn.scheduler.capacity.<queue-path>.state 

Description: The state of a queue (RUNNING or STOPPED). 

Value: RUNNING. 

yarn.scheduler.capacity.<queue-path>.user-limit-factor 

Description: The percentage of the queue's capacity a single user can consume. 

Value: Floating-point number (e.g., 1.0 for 100%, allowing a single user to use the entire queue capacity). 

yarn.scheduler.capacity.<queue-path>.maximum-applications: 

Description: The maximum number of applications that can run concurrently in a queue. 

Value: Integer. Prevents a single queue from overwhelming the ResourceManager. 

yarn.scheduler.capacity.resource-calculator: 

Description: Specifies how resources are calculated (memory, or memory and CPU). 

Value: 

Memory vs. CPU: Modern workloads often demand a balance of CPU and memory. Use DominantResourceCalculator with CapacityScheduler to ensure jobs are scheduled based on the resources (CPU or Memory) they are most constrained by. 

JVM Overheads: When configuring spark.executor.memory for Spark applications (or similar for other frameworks), remember that the allocated YARN container memory (yarn.scheduler.minimum-allocation-mb) must be larger than the JVM heap size requested by the application. This accounts for JVM overhead (e.g., off-heap memory, garbage collection, stack space). A common practice is to allocate an additional 10-20% beyond the application's requested heap. 

Queue Hierarchy Design: For multi-tenant clusters, design a logical queue hierarchy in CapacityScheduler (e.g., root.dev, root.prod, root.data_science) to isolate workloads, manage resource guarantees, and apply different access policies. 

Small Containers Overhead: While smaller minimum-allocation-mb values offer more granularity, they increase scheduling overhead for the ResourceManager and lead to more JVM instances, consuming more memory for JVM processes themselves. Balance granularity with efficiency. 

Dynamic Resource Allocation (Spark): For Spark, enabling spark.dynamicAllocation.enabled combined with spark.shuffle.service.enabled allows Spark to dynamically acquire and release executors from YARN based on workload, improving cluster utilization. Ensure YARN queues have sufficient maximum capacity for this to be effective. 

Monitoring: Continuously monitor YARN queue utilization, application resource consumption, and node health via the YARN ResourceManager UI (http://<rm_hostname>:8088) and cluster monitoring tools (e.g., Prometheus/Grafana, Ganglia) to fine-tune configurations. 

Preemption: CapacityScheduler supports preemption (yarn.scheduler.capacity.root.default.acl_submit_applications and related properties) which allows the scheduler to kill containers from over-capacity queues to free up resources for queues that are below their guaranteed capacity. This ensures fairness and resource guarantees.