2. Maria Colgan & Thierry Cruanes
Extreme Performance with Oracle Database 11g and In-Memory Parallel Execution
Maria Colgan & Thierry Cruanes

3. <Insert Picture Here>
Agenda
<Insert Picture Here>
Introduction to Parallel Execution
Automatic Degree Of Parallelism
Parallel Statement Queuing
In Memory Parallel Execution
Summary

4. Introduction to parallel execution

5. How Parallel Execution works
User
User connects to the database
Background process is spawned
When user issues a parallel SQL statement the background process becomes the Query Coordinator
Parallel servers communicate among themselves & the QC using messages that are passed via memory buffers in the shared pool
QC gets parallel servers from global pool and distributes the work to them
Parallel servers - individual sessions that perform work in parallel Allocated from a pool of globally available parallel server processes & assigned to a given operation
When a SQL statement is executed it will be hard parsed and a serial plan will be developed
The expected elapse time of that plan will be examined. If the expected Elapse time is
Less than PARALLEL_MIN_TIME_THRESHOLD then the query will execute serially.
If the expected Elapse time is greater than PARALLEL_MIN_TIME_THRESHOLD then the plan
Will be re-evaluated to run in parallel and the optimizer will determine the ideal DOP. The
Optimizer automatically determines the DOP based on the resource required for all scan operations
(full table scan, index fast full scan and so on)
However, the optimizer will cap the actual DOP for a statement with the default DOP
(paralllel_threads_per_cpu X CPU_COUNT X INSTANCE_COUNT), to ensure parallel
Processes do not flood the system.

6. Parallel Execution Plan
SELECT c.cust_name, s.purchase_date, s.amount
FROM sales s, customers c
WHERE s.cust_id = c.cust_id;
Query Coordinator ID
Operation
Name TQ
IN-OUT
PQ Distribution
SELECT STATEMENT                      1
  PX COORDINATOR 2
   PX SEND QC {RANDOM}                         
Q1,01
P->S 3
    HASH JOIN                      
PCWP 4
          PX RECEIVE             5
PX SEND BROADCAST               
P->P
BROADCAST 6
     PX BLOCK ITERATOR          7
      TABLE ACCESS FULL
CUSTOMERS 8
  PX BLOCK ITERATOR   9
     TABLE ACCESS FULL           
SALES
Parallel Servers
do majority of the work

7. Parallel Execution of a Query
SELECT c.cust_name, s.date, s.amount
FROM sales s, customers c
WHERE s.cust_id = c.cust_id;
Consumers
Producers

8. Producers and Consumer in the execution plan
Consumers
Query Coordinator ID
Operation
Name TQ
IN-OUT
PQ Distribution
SELECT STATEMENT                      1
  PX COORDINATOR 2
   PX SEND QC {RANDOM}                         
Q1,02
P->S 3
    HASH JOIN                      
PCWP 4
          PX RECEIVE             5
PX SEND HASH            
Q1,00
P->P 6
     PX BLOCK ITERATOR          7
      TABLE ACCESS FULL
CUSTOMERS 8
PX RECEIVE 9
PX SEND HASH
Q1,01 10
  PX BLOCK ITERATOR   11
     TABLE ACCESS FULL           
SALES
Producers

9. Parallel Execution of a Scan
Full scan of
the sales table
PQ 1
PQ 2
PQ 3
Data is divided into Granules
block range or partition
Each Parallel Server is assigned one or more Granules
No two Parallel Servers ever contend for the same Granule
Granules are assigned so that the load is balanced across all Parallel Servers
Dynamic Granules chosen by the optimizer
Granule decision is visible in execution plan

10. Identifying Granules of Parallelism during scans in the plan

11. Enabling Parallel Execution
There are three ways to enable parallel Execution
Enable the table(s) for parallel execution:
alter table sales parallel ;
alter table customers parallel ;
Use a parallel hint
select /*+ parallel(c) parallel(s) */
c.state_province, sum(s.amount) revenue
from customers c, sales s
where s.customer_id = c.id
and s.purchase_date=to_date('01-JAN-2007','DD-MON-YYYY')
and c.country = 'United States'
group by c.state_province;
Use alter session force parallel query ;

12. Controlling Parallel Execution on RAC
Use RAC Services
Create two services
Srvctl add service –d database_name
-s ETL
-r sid1, sid2
-s AHOC
-r sid3, sid4
Ad-Hoc queries
ETL
RAC was architected to allow database applications to be spread across lots of smaller servers allowing you to take advantage of the lower cost of commodity hardware to run business applications.
Oracle introduced services to help manage workload against a RAC database. DBA can choose where a service runs (IE which instance) and can dynamically change the allocation.
IF workload increases and more capacity is needed, simply add a new node to the cluster, a new instance to the RAC database.
In Oracle Database 11g, Oracle continues to improve the algorithms used to manage the shared cache in a RAC. This reduces the messaging required and improves performance. Applications where the workload is mostly read should see a performance boost with 11g.
Note:New Parameter to force a parallel statement to run on just node the query was issued on called PARALLEL_FORCE_LOCAL default FALSE

13. How could we enhance Parallel Execution?
Current Issues
Difficult to determine ideal DOP for each table without manual tuning
One DOP does not fit all queries touching an object
Not enough PX server processes can result in statement running serial
Too many PX server processes can thrash the system
Only uses IO resources
Solution
Oracle automatically decides if a statement
Executes in parallel or not and what DOP it will use
Can execute immediately or will be queued
Will take advantage of aggregated cluster memory or not
Prior to 11gR2 customers had to manual determine what DOP a statement should run with.
With no clear guide on how to determine the ideal DOP, customers would generally decide
on a DOP for an object (table, index) and all statements accessing that object would use
the same DOP. But one DOP may not be suitable for all statements.
The only other alternative was to use a different PARALLEL hint for each statement, but a
lot of third-party applications do not allow hints to be used and even if you could put a
hint for every statement it would mean a time consuming tuning exercise every time a
new statement got added to the system.
In 11gR2 Oracle automatically decides if a statement should execute in parallel or not
and what DOP it should use. Oracle will also decide if the statement can execute
immediately or if it should be queued until more system resources are available.
Finally Oracle will decide if the statement can take advantage of the aggregated cluster
memory or not (IN-Memory PQ)

14. Automatic Degree Of Parallelism

15. Automatic Degree of Parallelism
Business Requirement
Parallelism is completely manual
Tuning typically required to determine ideal DOP
Generally reserved for well-defined workload(large SQL)
One DOP does not fit all queries touching an object
Solution
Oracle automatically decides if a statement
Executes in parallel or not
What DOP the statement will use
Prior to 11gR2 customers had to manual determine what DOP a statement should run with.
With no clear guide on how to determine the ideal DOP, customers would generally decide
on a DOP for an object (table, index) and all statements accessing that object would use
the same DOP. But one DOP may not be suitable for all statements.
The only other alternative was to use a different PARALLEL hint for each statement, but a
lot of third-party applications do not allow hints to be used and even if you could put a
hint for every statement it would mean a time consuming tuning exercise every time a
new statement got added to the system.
In 11gR2 Oracle automatically decides if a statement should execute in parallel or not
and what DOP it should use. Oracle will also decide if the statement can execute
immediately or if it should be queued until more system resources are available.
Finally Oracle will decide if the statement can take advantage of the aggregated cluster
memory or not (IN-Memory PQ)

16. Automatic Degree of Parallelism Auto DOP
Statement with elapse times less than the threshold go serial
Statement with elapse times greater than threshold are candidates for Parallel
Optimizer derives the DOP for the statement based on resource requirements for all scans operations
Applies to all types of statements Query, DML, or DDL
Explain plan has been enhanced to show DOP selected
When automatic degree of parallelism (Auto DOP) is on the Oracle Optimizer will
Automatically decide the DOP for a statement based on the resource requirements
of the statement. Any statement that can be parallelized is a candidate for AUTO DOP.
You can see the DOP that the optimizer came up with in the notes section of the
explain plan (see below)
There are two init.ora parameters that control auto DOP
PARALLEL_DEGREE_POLICY and PARALLEL_MIN_TIME_THRESHOLD.
PARALLEL_DEGREE_POLICY controls whether or not auto DOP will be used.
By default Auto DOP is switch off for backward compatibility (MANUAL). To
Enable Auto DOP for everything set the parameter to AUTO. It is recommended
that DW users choose the LIMITED setting. It will apply auto DOP only for statements
where at least one table is decorated with PARALLEL clause.
AUTO_DEGREE_POLICY:
o        MANUAL - reverts to Oracle Database 10g behavior (Default).
o        LIMITED - auto DOP applied only to stmts that contain tables or indexes decorated explicitly with the parallel clause with or without explicit DOP
o        AUTO - automatic PQ for all statements.
PARALLEL_MIN_TIME_THREADHOLD: the execution time, as estimated by the optimizer,
above which a statement is considered for automatic PQ and automatic derivation of DOP.
By default this is set to AUTO which means 30sec.

17. How Auto Degree of Parallelism works
Statement is hard parsed
And optimizer determines the execution plan
SQL statement
Optimizer determines ideal DOP
If estimated time greater than threshold
Statement executes serially
If estimated time less than threshold
Statement executes in parallel
Actual DOP = MIN(PARALLEL_DEGREE_LIMIT, ideal DOP)
When a SQL statement is executed it will be hard parsed and a serial plan will be developed
The expected elapse time of that plan will be examined. If the expected Elapse time is
Less than PARALLEL_MIN_TIME_THRESHOLD then the query will execute serially.
If the expected Elapse time is greater than PARALLEL_MIN_TIME_THRESHOLD then the plan
Will be re-evaluated to run in parallel and the optimizer will determine the ideal DOP. The
Optimizer automatically determines the DOP based on the resource required for all scan operations
(full table scan, index fast full scan and so on)
However, the optimizer will cap the actual DOP for a statement with the default DOP
(paralllel_threads_per_cpu X CPU_COUNT X INSTANCE_COUNT), to ensure parallel
Processes do not flood the system.

18. Controlled by two init.ora parameters: PARALLEL_DEGREE_POLICY
Controlling Auto DOP
Controlled by two init.ora parameters:
PARALLEL_DEGREE_POLICY
Controls whether or not auto DOP will be used
Default is MANUAL which means no Auto DOP
Set to AUTO to enable auto DOP
PARALLEL_MIN_TIME_THRESHOLD
controls which statements are candidate for parallelism
Default is 10 seconds

19. Statement above threshold are candidate for parallelism
Controlling Auto DOP
Statement with an elapse time less than PARALLEL_MIN_TIME_THRESHOLD run serial
Statement above threshold are candidate for parallelism
Maximum DOP controlled by PARALLEL_DEGREE_LIMIT
Default value is CPU
PARALLEL_THREADS_PER_CPU X CPU_COUNT
Actual DOP = MIN(PARALLEL_DEGREE_LIMIT, ideal DOP)
Statements excluded from auto-dop
SQL issued from PL/SQL system packages
Index rebuild, statistics collection etc
Typically PL/SQL packages control their own parallelism
Note that a costly statement may go serial if none of its operations are expensive enough or the statement’s execution plan won’t scale

20. Explain plan enhancement for Auto DOP
PLAN_TABLE_OUTPUT
Plan hash value: ID
Operation
Name
Rows
Bytes
Cost
Time
Pstart
Pstop
Select Statement
96000
9889 5
00:00:01 1
PX COORDINATOR 2
PX SEND QC (RANDOM)
:TQ100 3
PX BLOCK ITERATOR 4
Table Access Full
Sales 16
You can see the DOP that the optimizer came up with in the notes section under the
explain plan (see below)
PLAN_TABLE_OUTPUT
Plan hash value:
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Pstart| Pstop |
| 0 | SELECT STATEMENT | | | | (0)| 00:00:01 | | |
| 1 | PARTITION RANGE ALL| | | | (0)| 00:00:01 | | |
| 2 | TABLE ACCESS FULL | SALES | | | 5 (0)| 00:00:01 | | |
Note
-----
- Computed Degree of Parallelism is 1 because of parallel threshold
Note
- Computed Degree of Parallelism is 16 because of parallel threshold

21. Parallel Statement Queuing

22. Parallel Statement Queuing
Business Requirement
With the introduction of Auto DOP
More statements will run in parallel
Possible to exhaust all parallel execution server processes
Potential system thrashing due to too many processes
Solution
Parallel Statement Queuing
Oracle automatically decides if a statement can execute immediately or not
Prevents serializing parallel queries when parallel servers are not available
Prevents system thrashing
Prior to 11gR2 customers had to manual determine what DOP a statement should run with.
With no clear guide on how to determine the ideal DOP, customers would generally decide
on a DOP for an object (table, index) and all statements accessing that object would use
the same DOP. But one DOP may not be suitable for all statements.
The only other alternative was to use a different PARALLEL hint for each statement, but a
lot of third-party applications do not allow hints to be used and even if you could put a
hint for every statement it would mean a time consuming tuning exercise every time a
new statement got added to the system.
In 11gR2 Oracle automatically decides if a statement should execute in parallel or not
and what DOP it should use. Oracle will also decide if the statement can execute
immediately or if it should be queued until more system resources are available.
Finally Oracle will decide if the statement can take advantage of the aggregated cluster
memory or not (IN-Memory PQ)

23. Parallel Statement Queuing
When a parallel statement starts checks if PX servers are available
Let it run if there are enough PX servers available
Queue the statement if there are not enough PX servers available
Monitors RAC-wide availability of PX servers
Adaptive to dynamic environments
Services
The service your session belongs to determines the limits on queuing
Cluster reconfiguration
Queue is aware of nodes leaving and joining the cluster and adjusts the limits accordingly
With the new auto DOP feature in 11.2 if it is left unrestrained it possible we could
Over load a system with parallel queries which could cause the CPU
To trash or worse still the database could run out of parallel server processes
And some of the parallel queries would have to run serial. In order to prevent
Both of these negative side effects we have introduce statement queuing in 11.2.
With statement queuing all serial statements will execute immediately but before a
Parallel statement is executed Oracle will do a quick check to see if there are enough
Parallel server processes available on the system. If there are then the parallel stmt
Can execute immediately but if there are not enough processes then the parallel
Statement is queued.
The queue is a strictly enforced First In First Out (FIFO) queue to ensure no parallel
Statement is starved out.
The parameter PARALLEL_INSTANCE_TARGET controls how many parallel server
Processes have to be available to prevent a parallel statement from being queued ??????

24. How Parallel Statement Queuing Works
Statement is parsed
and oracle automatically determines DOP
SQL statements
If not enough parallel servers available queue the statement
When the required number of parallel servers become available the first stmt on the queue is dequeued and executed
128 16 32 64
128 16 32 64
FIFO Queue
If enough parallel servers available execute immediately 8
Step 1: SQL Statement is issued and Oracle automatically determines if
It will run in parallel and if so what DOP it will get.
Step 2: Oracle checks to see if there are enough PQ servers to execute
The query. If there are it will execute immediately if there are not then
The query will be queued in a FIFO
Step 3: Everyone waits in the queue for the statement at the top of the
Queue to get all of his requested PQ servers even if there are enough
PQ servers available for one of the other statements to run ??????
Step 4: When enough PQ servers are available the statement is
De-queued and allowed to execute.

25. Controlling Parallel Statement Queuing
Enabled when PARALLEL_DEGREE_POLICY is set to AUTO
The Statement queue is enforced with a strict FIFO policy
PARALLEL_SERVER_TARGET indicates how many PX servers are available to run queries before queuing kicks-in
Default values 4 X PARALLEL_THREADS_PER_CPU X CPU_COUNT
This a soft limit and does not replace PARALEL_MAX_SERVERS
Total PX servers available 8 64
160
On an 8 CPU system
CPU Count
Parallel Server Target
Parallel Max Server
PX server available
to run queries before
queuing kicks in

26. Controlling Parallel Statement Queuing
Two new hints
To by-passes parallel statement queuing
SELECT /*+ NO_STMT_QUEUING */ col1 FROM foo;
To delay statement execution until resources are available without having PARALLEL_DEGREE_POLICY is set to AUTO
SELECT /*+ STMT_QUEUING */ col1 FROM foo;
V$SQL_PLAN_MONITOR has a new status value for SQL that is queued
SELECT s.sql_id, s.sql_text
FROM v$SQL_MONITOR m, v$SQL s
WHERE m.status='QUEUED’
AND m.sql_id = s.sql_id;
Two new wait events
PX Queuing: Statement queue
Indicates the first query in the queue
ENQ JX SQL statement queue
All other queries in the queue wait on this enqueue

27. Monitoring Statement Queuing in EM
Click on the SQL ID for more info
Awaiting screen shot from EM
Clock symbol indicated a queued statement

28. Monitoring Statement Queuing in EM
Wait event indicates stmt is at the head of the queue

29. Monitoring Statement Queuing in EM
Wait event indicates stmt is queued

30. In-Memory Parallel Execution

31. In-Memory Parallel Execution
Business Requirement
Traditionally Parallel Execution takes advantage of the IO capacity of a system
Disk speeds are not keeping up with Moore’s law while CPU and Memory are
Solution
In-Memory Parallel Execution harness the memory capacity of the entire system
Scan data nearly 10 X faster than scanning from disk
Prior to 11gR2 customers had to manual determine what DOP a statement should run with.
With no clear guide on how to determine the ideal DOP, customers would generally decide
on a DOP for an object (table, index) and all statements accessing that object would use
the same DOP. But one DOP may not be suitable for all statements.
The only other alternative was to use a different PARALLEL hint for each statement, but a
lot of third-party applications do not allow hints to be used and even if you could put a
hint for every statement it would mean a time consuming tuning exercise every time a
new statement got added to the system.
In 11gR2 Oracle automatically decides if a statement should execute in parallel or not
and what DOP it should use. Oracle will also decide if the statement can execute
immediately or if it should be queued until more system resources are available.
Finally Oracle will decide if the statement can take advantage of the aggregated cluster
memory or not (IN-Memory PQ)

32. Prior to Oracle Database 11gR2 Parallel Execution and the buffer cache

33. Prior to Oracle Database 11gR2 Parallel Execution and the buffer cache

34. How In-Memory Parallel Execution Works
Detect if the object fits in the aggregated buffer cache of the cluster
If so, distribute & affinitizes the blocks among the nodes and make PQ aware of the affinity
If not, continue to by-pass the buffer cache and read directly from disk
Subsequent access to the object will be conducted only by PX servers on the node to each the data was affinitized
FYI, the underscore parameter _Parallel_cluster_cache_policy can be used to turn
off cache affinity but it is on by default when PARALLEL_DEGREE_POLICY is on.

35. In-Memory Parallel Execution

36. How In-Memory Parallel Execution Works in detail
Decision to use the buffer cache is based on set of heuristics including
Ratio between buffer cache size and object size
Frequency at which the object is accessed
How much the object changes between accesses
In RAC fragments of the object are affinitized in the buffer cache on each of the active instances
Affinity is based on FileNumber and ExtentNumber unless hash partitioned
Automatically prevents multiple instances reading the same data from disk
Only PX process on the same RAC node can access each of the fragments of the object
FYI, the underscore parameter _Parallel_cluster_cache_policy can be used to turn
off cache affinity but it is on by default when PARALLEL_DEGREE_POLICY is on.

37. How In-Memory Parallel Execution Works
Determine the size of the table being looked at
Table is a good candidate for In-Memory Parallel Execution
Fragments of Table are read into each node’s buffer cache
Only parallel server on the same RAC node will access each fragment
SQL statement
Read into the buffer cache on any node
Table is extremely small
Table is extremely Large
Always use direct read from disk

38. Controlling In-Memory Parallel Execution
Controlled by PARALLEL_DEGREE_POLICY
Active only when set to AUTO
No way to turn it off

39. Summary

40. New Parallel Execution Init.ora Parameters
Value
Description
PARALLEL_DEGREE_LIMIT
CPU
Max DOP that can be selected with AUTO DOP
PARALLEL_DEGREE_POLICY
MANUAL
Specifies if AUTO DOP, Queuing, & In-memory PE will be enabled
PARALLEL_FORCE_LOCAL
FALSE
Restricts parallel server processes to the node where query is issued
PARALLEL_MIN_TIME_THRESHOLD
AUTO
Specifies min execution time a statement should have before AUTO DOP will kick in
PARALLEL_SERVERS_TARGET
4*CPU_COUNT* PARALLEL_THREADS_PER_CPU * ACTIVE_INSTANCES
Specifies # of parallel processes allowed to run parallel stmts before queuing will be use
PARALLEL_EXECUTION_MESSAGE_SIZE
16KB
Specifies size of the message buffers used for communication

41. Gradually introduce Auto Parallel Execution
PARALLEL_DEGREE_POLICY has three possible modes
Manual
As before, DBA must manually specify all aspects of parallelism
No AUTO DOP, Stmt Queuing, In-Memory Parallel Execution
Useful for well-understood existing applications
Limited
Restricted AUTO DOP for queries with tables decorated with default PARALLEL
No Stmt Queuing, In-Memory Parallel Execution
Useful in a mixed-world environment when a limited number of statements would benefit from parallel execution
Auto
All qualified statements subject to executing in parallel
Statements can be queued
IN-memory PQ available
Useful when deploying new applications in 11g that would benefit from parallel execution
Prior to 11gR2 customers had to manual determine what DOP a statement should run with.
With no clear guide on how to determine the ideal DOP, customers would generally decide
on a DOP for an object (table, index) and all statements accessing that object would use
the same DOP. But one DOP may not be suitable for all statements.
The only other alternative was to use a different PARALLEL hint for each statement, but a
lot of third-party applications do not allow hints to be used and even if you could put a
hint for every statement it would mean a time consuming tuning exercise every time a
new statement got added to the system.
In 11gR2 Oracle automatically decides if a statement should execute in parallel or not
and what DOP it should use. Oracle will also decide if the statement can execute
immediately or if it should be queued until more system resources are available.
Finally Oracle will decide if the statement can take advantage of the aggregated cluster
memory or not (IN-Memory PQ)

42. search.oracle.com For More Information Parallel Execution Or

43. The preceding is intended to outline our general product direction
The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.