GPU Scheduling on the NVIDIA TX2:

GPU Scheduling on the NVIDIA TX2:
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work GPU Scheduling on the NVIDIA TX2: Hidden Details Revealed Tanya Amert, Nathan Otterness, Ming Yang, James H. Anderson, F. Donelson Smith University of North Carolina at Chapel Hill

Do we have any guarantees about execution order?
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Motivation Do we have any guarantees about execution order? Climate Control Steering Turtle Detection

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work The Challenge Size, weight, and power (SWaP) constraints require embedded computing platforms, which limit processing power. We must keep utilization as high as possible.

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work The Challenge Size, weight, and power (SWaP) constraints require embedded computing platforms, which limit processing power. We must keep utilization as high as possible. NVIDIA GPUs are treated as black boxes, but used in safety-critical applications. These devices must be certified, but we need a model for GPU execution that allows concurrent execution.

Outline Motivation CUDA Fundamentals GPU Scheduling Rules
Basic Rules Extended Rules Motivation Future Work Outline Motivation CUDA Fundamentals GPU Scheduling Rules Extensions to Rules Future work

CUDA Programming Model
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work CUDA Programming Model A CUDA program’s 5 steps: Allocate GPU memory Copy data from CPU to GPU Launch kernel Copy results from GPU to CPU Free GPU memory input output CPU GPU

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work CUDA Programming Model A CUDA program’s 5 steps: Allocate GPU memory Copy data from CPU to GPU Launch kernel Copy results from GPU to CPU Free GPU memory CPU GPU

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work CUDA Programming Model A GPU program launches a kernel. Kernels are specified by the number of thread blocks, and the threads per block. Kernel Blocks Threads

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work CUDA Programming Model Kernels are processed SIMD Each thread acts on different data Threads determine data using blockDim, blockIdx, threadIdx Note: a GPU thread is not an OS thread! We’ll call OS threads “tasks” __global__ void vecAdd(int *A, int *B, int *C) { int i = blockDim.x * blockIdx.x + threadIdx.x; C[i] = A[i] + B[i]; } Kernel Blocks Threads

Ordering of GPU Operations
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Ordering of GPU Operations CUDA operations can be ordered by associating them with a stream. A stream is a FIFO queue of operations. But that’s all that NVIDIA tells us… Questions: Can GPU operations in different streams run concurrently? They “may”… How are GPU operations from different streams ordered? How do streams differ? Default to a single stream, the NULL stream.

Non-Goals vs. Goals We are not trying to: certify GPUs.
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Non-Goals vs. Goals We are not trying to: certify GPUs. perform timing analysis of GPU-using systems. improve utilization by modifying scheduling behavior. Our goal is to: discover rules of GPU scheduling needed to build a model for GPU execution.

Scheduling Rules Questions:
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Scheduling Rules Questions: Can GPU operations in different streams run concurrently? They “may”… How are GPU operations from different streams ordered? How do streams differ? Default to a single stream, the NULL stream. Goal: Provide rules governing GPU scheduling behavior. Consider only CPU tasks within one address space. Focus on user-defined streams.

Experimental Setup – NVIDIA Jetson TX2
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experimental Setup – NVIDIA Jetson TX2 Kernels are executed on the execution engine (EE). The EE is made up of multiple streaming multiprocessors (SMs). Two SMs form EE One CE GPU programs also submit copy operations to the GPU. Copies are performed on a copy engine (CE).

Experimental Setup – Schedule Visualizations
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experimental Setup – Schedule Visualizations 2048 total GPU threads available on each SM 128 cores per SM

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experimental Setup – Schedule Visualizations Labeled by stream 1024 threads in block Arrow gives the time a kernel was submitted to the GPU Block start Block completion K1: 1 x 1024

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experimental Setup – Schedule Visualizations These simple kernels spin for a configurable amount of time – guarantees consistent runtime K1: 2 x 1024

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experimental Setup – Schedule Visualizations We say these blocks are assigned to the GPU Multiple blocks can run on a SM at one time K1: 3 x 1024

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experimental Setup – Schedule Visualizations K1 is dispatched when at least one block is assigned K1 is now fully dispatched, as all blocks have been assigned K1: 6 x 1024

Experiment #1: single stream
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #1: single stream Kernels in the same stream should execute in FIFO order.

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #1: single stream K1: 0 K1: 2 K1: 1 K1: 3 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Kernels in the same stream should execute in FIFO order. Let’s try it! K1: 6 x 1024 K2: 2 x 512 It might not be clear to the audience how the config is working – this is a good opportunity to sell our open-source framework as well The dotted line indicates the time of the queue snapshot

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #1: single stream K1: 4 K1: 5 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Kernels in the same stream should execute in FIFO order. Let’s try it! K1: 6 x 1024 K2: 2 x 512

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #1: single stream K2: 0 K2: 1 GPU SM 0 SM 1 EE Queue Stream S1 K2 Kernels in the same stream should execute in FIFO order. Let’s try it! K1: 6 x 1024 K2: 2 x 512

Experiment #2: multiple streams
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #2: multiple streams What if we submit kernels from multiple streams? The documentation says they may run concurrently…

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #2: multiple streams K1: 0 K1: 2 K1: 1 K1: 3 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 What if we submit kernels from multiple streams? The documentation says they may run concurrently… K1: 6 x 1024 K2: 2 x 512 K3: 2 x 512

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #2: multiple streams K1: 0 K1: 2 K1: 1 K1: 3 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 K3 What if we submit kernels from multiple streams? The documentation says they may run concurrently… K1: 6 x 1024 K2: 2 x 512 K3: 2 x 512

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #2: multiple streams K1: 4 K3: 0 K1: 5 K3: 1 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 K3 What if we submit kernels from multiple streams? The documentation says they may run concurrently… K1: 6 x 1024 K2: 2 x 512 K3: 2 x 512 K3 is dispatched before K2 because it is earlier in the EE queue

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #2: multiple streams K1: 4 K3: 0 K1: 5 K3: 1 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 K3 What if we submit kernels from multiple streams? The documentation says they may run concurrently… K1: 6 x 1024 K2: 2 x 512 K3: 2 x 512 K3 must have entered the EE queue while K1 was running

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #2: multiple streams K2: 0 K2: 1 GPU SM 0 SM 1 EE Queue Stream S1 K2 Stream S2 What if we submit kernels from multiple streams? The documentation says they may run concurrently… K1: 6 x 1024 K2: 2 x 512 K3: 2 x 512 K2 does not enter the EE queue until K1 completes

Experiment #3: cut-ahead?
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #3: cut-ahead? Can a kernel cut ahead of a partially-dispatched kernel?

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #3: cut-ahead? K1: 0 K1: 2 K1: 1 K1: 3 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 Can a kernel cut ahead of a partially- dispatched kernel? K1: 6 x 768 K2: 2 x 512 K3: 2 x 512 K3 fits here

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #3: cut-ahead? K1: 0 K1: 2 K1: 1 K1: 3 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 K3 Can a kernel cut ahead of a partially- dispatched kernel? K1: 6 x 768 K2: 2 x 512 K3: 2 x 512 K3 fits here, but isn’t dispatched yet…

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #3: cut-ahead? Blocks 1,3 finished before 0,2 K3: 0 K3: 1 K1: 4 K1: 5 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 K3 Can a kernel cut ahead of a partially- dispatched kernel? K1: 6 x 768 K2: 2 x 512 K3: 2 x 512

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #3: cut-ahead? K3: 0 K3: 1 K1: 4 K1: 5 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 K3 Can a kernel cut ahead of a partially- dispatched kernel? K1: 6 x 768 K2: 2 x 512 K3: 2 x 512 Let’s take a step back in time…

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #3: cut-ahead? K1: 0 K1: 2 K1: 1 K1: 3 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 K3 Can a kernel cut ahead of a partially- dispatched kernel? K1: 6 x 768 K2: 2 x 512 K3: 2 x 512 K3 K1

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #3: cut-ahead? K3: 0 K3: 1 K1: 4 K1: 5 GPU SM 0 SM 1 EE Queue Stream S1 K1 K2 Stream S2 K3 Can a kernel cut ahead of a partially- dispatched kernel? K1: 6 x 768 K2: 2 x 512 K3: 2 x 512

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #3: cut-ahead? K2: 0 K2: 1 GPU SM 0 SM 1 EE Queue Stream S1 K2 Stream S2 Can a kernel cut ahead of a partially- dispatched kernel? K1: 6 x 768 K2: 2 x 512 K3: 2 x 512

Resource Requirements (4)
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Rules So Far General (4) G1: Kernels are enqueued on the associated stream queue. G2: A kernel is enqueued on the EE queue when it reaches the head of its stream queue. G3: A kernel at the head of the EE queue is dequeued from the EE queue when it becomes fully dispatched. G4: A kernel is dequeued from its stream queue once all of its blocks complete execution. X1: Only blocks of the kernel at the head of the EE queue are eligible to be assigned. R1: A block of the kernel at the head of the EE queue is eligible to be assigned only if its resource constraints are met. R2: A block of the kernel at the head of the EE queue is eligible to be assigned only if there are sufficient thread resources available on some SM. R3: A block of the kernel at the head of the EE queue is eligible to be assigned only if there are sufficient shared-memory resources available on some SM. Resource Requirements (4)

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Rules So Far General (4) Copy Operations (4) G1: Kernels are enqueued on the associated stream queue. G2: A kernel is enqueued on the EE queue when it reaches the head of its stream queue. G3: A kernel at the head of the EE queue is dequeued from the EE queue when it becomes fully dispatched. G4: A kernel is dequeued from its stream queue once all of its blocks complete execution. X1: Only blocks of the kernel at the head of the EE queue are eligible to be assigned. R1: A block of the kernel at the head of the EE queue is eligible to be assigned only if its resource constraints are met. R2: A block of the kernel at the head of the EE queue is eligible to be assigned only if there are sufficient thread resources available on some SM. R3: A block of the kernel at the head of the EE queue is eligible to be assigned only if there are sufficient shared-memory resources available on some SM. C1: A copy operation is enqueued on the CE queue when it reaches the head of its stream queue. C2: A copy operation at the head of the CE queue is eligible to be assigned to the CE. C3: A copy operation at the head of the CE queue is dequeued from the CE queue once the copy is assigned to the CE on the GPU. C4: A copy operation is dequeued from its stream queue once the CE has completed the copy. Resource Requirements (4)

Full Experiment (see paper)
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Full Experiment (see paper)

Basic Rules Extended Rules Motivation Future Work Outline Motivation CUDA Fundamentals GPU Scheduling Rules Extensions to Rules Prioritized streams NULL stream Future work

Experiment #4: low-priority starvation
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: low-priority starvation Can high priority kernels starve low-priority kernels?

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: low-priority starvation K1: 0 K1: 2 K1: 1 K1: 3 GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 K1 Stream S2 K2 High-Pri EE Queue Stream S3 K3 Can high priority kernels starve low-priority kernels? Low: K1: 8 x 1024 High: K2: 4 x 1024 K3: 4 x 1024

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: low-priority starvation K2: 0 K2: 3 K2: 1 K2: 2 GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 K1 Stream S2 K2 High-Pri EE Queue Stream S3 K3 Can high priority kernels starve low-priority kernels? Low: K1: 8 x 1024 High: K2: 4 x 1024 K3: 4 x 1024 K3

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: low-priority starvation K3: 0 K3: 3 K3: 1 K3: 2 GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 K1 Stream S2 High-Pri EE Queue Stream S3 K3 Can high priority kernels starve low-priority kernels? Low: K1: 8 x 1024 High: K2: 4 x 1024 K3: 4 x 1024

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: low-priority starvation K1: 5 K1: 7 K1: 4 K1: 6 GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 K1 Stream S2 High-Pri EE Queue Stream S3 Can high priority kernels starve low-priority kernels? Low: K1: 8 x 1024 High: K2: 4 x 1024 K3: 4 x 1024 K1 is starved by multiple higher-priority streams’ kernels

Experiment #5: NULL stream
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #5: NULL stream How does the NULL stream interact with user-defined kernels?

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #5: NULL stream GPU SM 0 SM 1 Stream S1 Stream S2 EE Queue NULL Stream How does the NULL stream interact with user-defined kernels? User-defined streams and the NULL stream feed into one EE queue

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #5: NULL stream K1: 0 K1: 1 GPU SM 0 SM 1 Stream S1 K1 Stream S2 EE Queue NULL Stream How does the NULL stream interact with user-defined kernels? K1: 2 x K3: 2 x 1024 NULL: K2: 1 x 1024

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #5: NULL stream K1: 0 K1: 1 GPU SM 0 SM 1 Stream S1 K1 Stream S2 EE Queue NULL Stream K3 K2 How does the NULL stream interact with user-defined kernels? K1: 2 x K3: 2 x 1024 NULL: K2: 1 x 1024 K2 fits here, but isn’t dispatched yet… K3 is not in the EE queue

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #5: NULL stream K2: 0 GPU SM 0 SM 1 Stream S1 Stream S2 EE Queue NULL Stream K3 K2 How does the NULL stream interact with user-defined kernels? K1: 2 x K3: 2 x 1024 NULL: K2: 1 x 1024

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #5: NULL stream K3: 0 K3: 1 GPU SM 0 SM 1 Stream S1 Stream S2 EE Queue NULL Stream K3 How does the NULL stream interact with user-defined kernels? K1: 2 x K3: 2 x 1024 NULL: K2: 1 x 1024 K1 and K3 could have executed concurrently

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Extended Rules General (4) Copy Operations (4) G1: Kernels are enqueued on the associated stream queue. G2: A kernel is enqueued on the EE queue when it reaches the head of its stream queue. G3: A kernel at the head of the EE queue is dequeued from the EE queue when it becomes fully dispatched. G4: A kernel is dequeued from its stream queue once all of its blocks complete execution. X1: Only blocks of the kernel at the head of the EE queue are eligible to be assigned. R1: A block of the kernel at the head of the EE queue is eligible to be assigned only if its resource constraints are met. R2: A block of the kernel at the head of the EE queue is eligible to be assigned only if there are sufficient thread resources available on some SM. R3: A block of the kernel at the head of the EE queue is eligible to be assigned only if there are sufficient shared-memory resources available on some SM. C1: A copy operation is enqueued on the CE queue when it reaches the head of its stream queue. C2: A copy operation at the head of the CE queue is eligible to be assigned to the CE. C3: A copy operation at the head of the CE queue is dequeued from the CE queue once the copy is assigned to the CE on the GPU. C4: A copy operation is dequeued from its stream queue once the CE has completed the copy. N1: A kernel Kk at the head of the NULL stream queue is enqueued on the EE queue when, for each other stream queue, either that queue is empty or the kernel at its head was launched after Kk. N2: A kernel Kk at the head of a non-NULL stream queue cannot be enqueued on the EE queue unless the NULL stream queue is either empty or the kernel at its head was launched after Kk. A1: A kernel can only be enqueued on the EE queue matching the priority of its stream. A2: A block of a kernel at the head of any EE queue is eligible to be assigned only if all higher-priority EE queues (priority-high over priority- low) are empty. Resource Requirements (4) NULL Stream (2) Prioritized Steams (2)

An API call caused the GPU to wait for a synchronization point
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Future Work We plan to extend our rules to include more complex behavior and explore sources of implicit synchronization. Why didn’t K3 and K4 run? An API call caused the GPU to wait for a synchronization point

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Future Work We plan to extend our rules to include more complex behavior and explore sources of implicit synchronization. Our rules will lead to a new model for GPU program execution. Add citation to RTAS paper

Summary Contributions: Rules for GPU Execution
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Summary Contributions: Rules for GPU Execution Extended experimentation framework Next steps: Extend the rules for complex scenarios Investigate synchronization effects cuda_scheduling_examiner_mirror

Experiment #4: prioritized streams
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: prioritized streams What happens with streams of different priorities?

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: prioritized streams GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 Stream S2 High-Pri EE Queue What happens with streams of different priorities? There are multiple EE queues, one per priority level If not specified, the default is low

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: prioritized streams GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 Stream S2 High-Pri EE Queue K1: 0 K1: 2 K1: 1 K1: 3 GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 K1 Stream S2 K2 High-Pri EE Queue What happens with streams of different priorities? Low: K1: 8 x 1024 High: K2: 2 x 1024

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: prioritized streams K1: 4 K1: 5 K2: 0 K2: 1 GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 K1 Stream S2 K2 High-Pri EE Queue What happens with streams of different priorities? Low: K1: 8 x 1024 High: K2: 2 x 1024 K1 is preempted (between blocks) by K2

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Experiment #4: prioritized streams K1: 7 K2: 6 GPU SM 0 SM 1 Low-Pri EE Queue Stream S1 K1 Stream S2 High-Pri EE Queue What happens with streams of different priorities? Low: K1: 8 x 1024 High: K2: 2 x 1024

NULL Stream Rules N1: NULL stream kernels wait for prior other kernels
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work NULL Stream Rules N1: NULL stream kernels wait for prior other kernels N2: Non-NULL stream kernels wait for NULL stream kernels Waiting here means not put on EE queue Change to all be one block size, as few blocks as possible K3 would have fit here, but K2 is a NULL stream kernel

NULL Stream Rules N1: NULL stream kernels wait for prior other kernels
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work NULL Stream Rules N1: NULL stream kernels wait for prior other kernels N2: Non-NULL stream kernels wait for NULL stream kernels Waiting here means not put on EE queue K5 would have fit here, but NULL stream kernels cannot run concurrently with others

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work NULL Stream Rules N1: A kernel Kk at the head of the NULL stream queue is enqueued on the EE queue when, for each other stream queue, either that queue is empty or the kernel at its head was launched after Kk. N2: A kernel Kk at the head of a non-NULL stream queue cannot be enqueued on the EE queue unless the NULL stream queue is either empty or the kernel at its head was launched after Kk.

Prioritized Stream Rules
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Prioritized Stream Rules A1: EE queue matches priority level A2: GPU chooses from EE queues by priority There are multiple EE queues, one per priority level K1 is preempted (between blocks) by K2

Prioritized Stream Rules
CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Prioritized Stream Rules A1: EE queue matches priority level A2: GPU chooses from EE queues by priority There are multiple EE queues, one per priority level Infinite starvation -> unbounded response time K1 is starved by multiple higher-priority streams’ kernels

CUDA Fundamentals Basic Rules Extended Rules Motivation Future Work Extended Rules General (4) Copy Operations (4) G1: Kernels are enqueued on the associated stream queue. G2: A kernel is enqueued on the EE queue when it reaches the head of its stream queue. G3: A kernel at the head of the EE queue is dequeued from the EE queue when it becomes fully dispatched. G4: A kernel is dequeued from its stream queue once all of its blocks complete execution. X1: Only blocks of the kernel at the head of the EE queue are eligible to be assigned. R1: A block of the kernel at the head of the EE queue is eligible to be assigned only if its resource constraints are met. R2: A block of the kernel at the head of the EE queue is eligible to be assigned only if there are sufficient thread resources available on some SM. R3: A block of the kernel at the head of the EE queue is eligible to be assigned only if there are sufficient shared-memory resources available on some SM. C1: A copy operation is enqueued on the CE queue when it reaches the head of its stream queue. C2: A copy operation at the head of the CE queue is eligible to be assigned to the CE. C3: A copy operation at the head of the CE queue is dequeued from the CE queue once the copy is assigned to the CE on the GPU. C4: A copy operation is dequeued from its stream queue once the CE has completed the copy. N1: A kernel Kk at the head of the NULL stream queue is enqueued on the EE queue when, for each other stream queue, either that queue is empty or the kernel at its head was launched after Kk. N2: A kernel Kk at the head of a non-NULL stream queue cannot be enqueued on the EE queue unless the NULL stream queue is either empty or the kernel at its head was launched after Kk. A1: A kernel can only be enqueued on the EE queue matching the priority of its stream. A2: A block of a kernel at the head of any EE queue is eligible to be assigned only if all higher-priority EE queues (priority-high over priority- low) are empty. Resource Requirements (4) NULL Stream (2) Prioritized Steams (2)

GPU Scheduling on the NVIDIA TX2:

Similar presentations

Presentation on theme: "GPU Scheduling on the NVIDIA TX2:"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

GPU Scheduling on the NVIDIA TX2:

Similar presentations

Presentation on theme: "GPU Scheduling on the NVIDIA TX2:"— Presentation transcript:

Similar presentations

About project

Feedback