1. Oracle Memory Configuration on Windows Server
Configuring Large Memory for Oracle on 32-bit and 64-bit Windows

2. Objectives
At the end of this module the student will understand the following concepts:
OS Memory Configuration
32-bit and 64-bit systems
Boot.ini file
/3GB, /PAE, and /NOPAE Switches
Oracle Memory Configuration
Awe enabled
Performance Implications of 32-bit Oracle
Advantages of 64-bit Oracle

3. 32-bit Addressing
32-bit processors have address registers that are 32-bits wide (can address up to 4GB physical RAM)
Allows per process virtual address space of 4GB range (0 to 4,294,967,295 bytes)
Each individual process gets its own virtual address space that can be up to 4GB
This virtual address space is divided – system and user space
The 4GB addressable space per process is divided into system and user space. By default, the system space is 2GB and the private user space is 2GB.
SYSTEM SHARED SPACE:
The system space is shared by all processes - so all processes are mapped to the same virtual addresses for the system portion. This allows processes to communicate.
PRIVATE USER SPACE:
Private addresses associated with that process only – this space can only be accessed by threads running in that process.

4. User and System Virtual Address Space
The 4 User blocks represent 4 user processes each with 2GB virtual address space. Each user process shares the same 2GB System address space.

5. 4GT (4GB Tuning) with /3GB
/3GB = 4GB Tuning Feature – reduces system space to 1GB, allowing 3GB for user space
Add to boot.ini file and reboot to take effect
For systems 4GB to 16GB only
The /3GB switch can be used on systems with 4GB RAM up to 16GB RAM.
This allows each user process to address up to 3GB (virtual address space) instead of the default of 2GB.
Systems with more than 16GB or more should NOT use /3GB – it will cause the memory above 16GB not to be used! Instead, for systems with more than 16GB RAM, remove the /3GB switch and include the /PAE switch only.

6. 32-bit Systems with PAE (36-bit)
PAE- Physical Address Extension
AKA Large Memory Support
Allows 36-bit memory addressing
Allows the IA32 processor to address up to 64GB of physical RAM
Add /PAE switch in the boot.ini file and reboot server
Adds translation layer to memory access
Inherently, 32-bit systems can address physical memory up to 4GB.
With PAE, physical memory above 4GB can be addressed (up to 64GB depending on the OS version), but there is memory mapping overhead with PAE that is not present in a 64-bit system.
With PAE, the OS adds a virtual memory translation layer to map 32-bit virtual addresses into the real 64GB address range.
32-bit applications are still limited to using a 32-bit virtual address space.

7. Windows 2003 /NOPAE Special Case (32-bit)
New for Windows 2003 is the /NOPAE switch
Forces Windows 2003 NOT to use PAE algorithms for memory access (there is overhead involved)
Should be used on systems with 4GB or less and hot-add memory devices
For Windows 2003 Enterprise and Datacenter Editions only:
If the server is using hot-add memory devices, then PAE is automatically enabled!
To disable it (you would want to if the system has 4GB or less to eliminate the extra layer of address translation), you must add /NOPAE in the boot.ini
If you’re not sure, it is safe to add /NOPAE for any Windows 2003 EE/DE system with 4GB or less.

8. Boot.ini Examples (32-bit)
To use /3GB or /PAE, add the switches to the boot.ini file and reboot the server
Note:
Using both /3GB and /PAE together conserves the most memory under 4 GB
Using both /3GB and /PAE will limit memory addressability to 16 GB
Example of boot.ini with /3GB and /PAE for a system with more than 4GB and <=16GB (32-bit only):
[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(2)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(2)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect /3GB /PAE
For systems with more than 16GB RAM, just use /PAE without /3GB.

9. 32-bit Memory Size/Switch Chart
Boot.ini SWITCH
< 4GB
None (or /NOPAE)
= 4GB
/3GB (or /3GB /NOPAE)
> 4GB & <= 16GB
/3GB /PAE
>16GB
/PAE only
This is for 32-bit systems only.

10. Support for /3GB and /PAE – 32-bit
Windows 2000 Advanced Server
Windows 2000 Datacenter Server
Windows 2003 Enterprise Edition
Windows 2003 Datacenter Edition
Only these versions of Windows fully support using the two switches in production systems.

11. 32-bit Max Memory Support on x86 based computers
Windows 2003 Enterprise = 32GB max
Windows 2003 Datacenter = 64GB max
You probably would not want to go above 32GB with a 32-bit Windows system because of the overhead of using PAE. If you need large memory over 16GB, definitely consider 64-bit Windows for direct memory addressing.

12. AWE for 32-bit Oracle Address Windowing Extension
AWE – a set of APIs that allow programs to address memory locations outside of their 4GB virtual addressing range
Available through 32-bit addressing
With AWE programs reserve memory as non-paged (will not page out)
AWE must be used in order for the Oracle program to use memory above 4GB (in conjunction with PAE).
AWE places the responsibility of virtual address translation on the application process (Oracle).
An in-process buffer area called the AWE region is defined and used to dynamically map allocated physical pages to 32-bit addresses.

13. Implementing AWE with 32-bit Oracle
Set the /PAE boot.ini flag
Set the “USE_INDIRECT_BUFFERS=TRUE” parameter in the Oracle initialization file.
Use the DB_BLOCK_BUFFERS parameter in the initialization file instead of the DB_CACHE parameter.
The AWE_WINDOW_SIZE registry parameter sets up a “swap area” in low memory to map data buffers in high memory
Default size is 1 GB.
Setting up AWE for Oracle just takes a few extra steps. First, the /PAE flag is set in the boot.ini file. In addition, two entries are edited in the init.ora file or spfile. These are the “USE_INDIRECT_BUFFERS=TRUE” parameter and the use of the DB_BLOCK_BUFFERS parameter instead of the DB_CACHE parameter. By default, a registry parameter – AWE_WINDOW_SIZE is set to map low memory to high memory. The default window size is 1 GB.

14. Performance Implications of 32-bit Memory Management
Memory split into two parts is not as efficient as memory in one segment
Overhead is introduced
32-bit systems with 8 GB of RAM may not perform much better than systems with 4 GB of RAM.
May need to add 12 GB of RAM to see a noticable return for the investment.
This assumes that the memory is being used for Oracle System Global Area.

15. Performance Implications of 32-bit Memory Management
Memory above 4 GB can only be used for DATA_BUFFER cache.
There is little room for the Shared Pool, Large Pool, PGA, etc.
The AWE_WINDOW_SIZE parameter takes up 1 GB of RAM
Leaves only 1 GB of RAM for all other memory types (2 GB with /3GB)
The other memory types are required for supporting memory connections.
As a result, Oracle on 32-bit systems is severely restricted on the number of user connections supported
100s – low thousands with Shared Servers
The way that Oracle allocates32-bit memory severely limits how many concurrent user connections can be supported.

16. Conserving Memory for User Connections
AWE/PAE does not improve user connectivity for 32-bit Oracle
User connections must share a limited pool of memory, 1-2 GB
The AWE_WINDOW_SIZE registry parameter can be decreased from 1 GB to as low as 512 MB to increase user memory
Tradeoff between upper memory accessibility and user memory size

17. Conserving Memory for User Connections
Oracle initialization parameters can be used to decrease per user memory usage
bitmap_merge_area_size
create_bitmap_area_size
hash_area_size
open_cursors
sort_area_size (sort_area_retained_size)
The Orastack command can be used to decrease memory use per thread
Default allocation is 1 MB
Risky to decrease below 700KB; any change must be tested
Example usage:
orastack oracle.exe
orastack tnslsnr.exe
orastack svrmgrl.exe
orastack sqlplus.exe

18. Windows bit Systems
64-bit versions of Windows have implemented up to 16 terabytes of virtual address space
Don’t use PAE or /3GB with 64-bit
64-bit programs use 16 terabyte tuning model (8TB user / 8TB kernel)
Compare to 32-bit tuning mode (2GB user / 2GB kernel)
No performance penalties for large memory
Unlimited user connectivity
64-bit Windows versions allow addressing of large physical memory sizes without using PAE.
64-bit Windows is supported on Intel Itanium based servers, Intel EM64T based Servers, and AMD Opteron based servers.
Only Windows 2003 Enterprise Edition and Datacenter are available for Itanium 64-bit.
Standard Edition, Enterprise Edition and Datacenter Edition are available for the EM64T based servers and Opteron based servers.

19. 64-bit Memory Support on Itanium based computers
Windows 2003 Enterprise 64-bit = 1TB max
Windows 2003 Datacenter 64-bit = 1TB max
64-bit addressing allows direct large memory addressing. The virtual address space per process currently supported by Windows 2003 allows 16 Terabytes of virtual address space – 8TB system/8TB user.

20. 64-bit Memory Support on EM64T and Opteron based computers
Windows 2003 Standard 64-bit = 64GB
Windows 2003 Enterprise 64-bit = 1TB max
Windows 2003 Datacenter 64-bit = 1 TB max
EM64T and Opteron based servers also have the advantage of supporting full native32-bit aplication support concurrent with full native 64-bit applications. Itanium2 servers can only emulate 32-bit support, at a far slower speed.

21. Monitor Oracle Memory Oracle for Windows Performance Counters
Oracle %phyreads/gets shows Buffer Cache performance
Should be below 10% - few physical reads as a proportion of all reads; otherwise, more memory may be required

22. Review
What flag is set in the boot.ini file to use memory above 16 GB?
Which two flags may be set in the boot.ini file for memory between 4 GB and 16 GB?
What are the performance implications of 32-bit memory management for Oracle?
What tool may be used to monitor Buffer Cache efficiency?

23. Summary The following topics were discussed in this training module:
OS Memory Configuration
32-bit and 64-bit systems
Boot.ini file
/3GB, /PAE, and /NOPAE Switches
Oracle Memory Configuration with AWE
Performance Implications of 32-bit Oracle
Advantages of 64-bit Oracle