Oracle Partitioning in Oracle Database 11g

Oracle Partitioning in Oracle Database 11g

Oracle Partitioning Ten Years of Development
Fast drop table “Multi-dimensional” pruning 1M partitions per table Oracle10g R2 Local Index maintenance Global hash indexes Oracle10g Fast partition split Composite range-list partitioning Oracle9i R2 Global index maintenance List partitioning Oracle9i Merge operation Partition-wise joins “Dynamic” pruning Hash and composite range-hash partitioning Oracle8i Basic maintenance operations: add, drop, exchange “Static” partition pruning Range partitioning Global range indexes Oracle8 Manageability Performance Core functionality #partitions: Yahoo already at 430,000

Oracle Partitioning Ten Years of Development
Interval Partitioning Partition Advisor More composite choices REF Partitioning Virtual Column Partitioning Oracle Database 11g Fast drop table “Multi-dimensional” pruning 1M partitions per table Oracle10g R2 Local Index maintenance Global hash indexes Oracle10g Fast partition split Composite range-list partitioning Oracle9i R2 Global index maintenance List partitioning Oracle9i Merge operation Partition-wise joins “Dynamic” pruning Hash and composite range-hash partitioning Oracle8i Basic maintenance operations: add, drop, exchange “Static” partition pruning Range partitioning Global range indexes Oracle8 Manageability Performance Core functionality Note: improvements in core functionality feed performance improvements, e.g. REF partitioning enables pruning and PWJ against child tables

Oracle Partitioning Fundamental Concepts for Success
While performance seems to be the most visible one, don't forget about the rest partitioning must address all business-relevant areas of Performance, Manageability, and Availability Partition autonomy is crucial fundamental requirement for any partition maintenance operations acknowledge partitions as metadata in the data dictionary Cf. microsoft where you can not even name a partition, let alone manage it autonomously. Provide full partition autonomy Use local indexes whenever possible Enable partition all table-level operations for partitions, e.g. TRUNCATE, MOVE, COMPRESS Make partitions visible and usable for database administration Partition naming for ease of use Maintenance operations must be partition-aware Also true for indexes Maintenance operations must not interfere with online usage of a partitioned table 4

Oracle Partitioning Enhancements
Complete the basic partitioning strategies defines HOW data is going to be partitioned new composite partitioning methods Introduce partitioning extensions defines WHAT controls the data placement enhance the manageability and automation virtual column based partitioning REF partitioning interval partitioning partition advisor

<Insert Picture Here>
Composite Partitioning in Oracle Database 11g

Extended Composite Partitioning Strategies
Concept of composite partitioning Data is partitioned along two dimensions (A,B)‏ A distinct value pair for the two dimensions uniquely determines the target partitioning Composite partitioning is complementary to multi- column range partitioning Extensions in Oracle Database 11g .. New 11g Strategy Use Case List – Range Geography -Time Range - Range ShipDate - OrderDate List - Hash Geography - OrderID List - List Geography - Product Cf. multi column range where second column is only used if first is non deterministic

Composite Partitioning - Concept
Table SALES RANGE(order_date)-RANGE(ship_date)‏ ship_date Jan 2006 ... ... Feb 2006 ... ... ... ... May 2006 ... ... ... ... Jan 2006 Feb 2006 Mar 2006 Jan 2007 order_date

Table SALES RANGE(order_date)-RANGE(ship_date)‏ ship_date Mar 2006 All records with order_date in March 2006 Jan 2006 ... ... Feb 2006 ... ... ... ... May 2006 ... ... ... ... Jan 2006 Feb 2006 Mar 2006 Jan 2007 order_date

Table SALES RANGE(order_date)-RANGE(ship_date)‏ ship_date All records with ship_date in May 2006 Jan 2006 ... ... Feb 2006 ... ... ... ... May 2006 May 2006 ... ... ... ... Jan 2006 Feb 2006 Mar 2006 Jan 2007 order_date

Table SALES RANGE(order_date)-RANGE(ship_date)‏ ship_date All records with order_date in March AND ship_date in May 2006 Jan 2006 ... ... Feb 2006 ... ... ... ... May 2006 May 2006 ... ... ... ... Mar 2006 Jan 2006 Feb 2006 Mar 2006 Jan 2007 order_date

Virtual Column based Partitioning

Virtual Columns Business Problem Solution
Extended Schema attributes are fully derived and dependent on existing common data Redundant storage or extended view definitions are solving this problem today requires additional maintenance and creates overhead Solution Oracle Database 11g introduces virtual columns purely virtual, meta-data only Treated as real columns except no DML can have statistics eligible as partitioning key Enhanced performance and manageability Esp. use when migrating from legacy where overloaded columns common virtual columns: - yes, you can enforce uniqueness (or even a PK on them) - yes, you can define referential integrity to point to them

Virtual Columns - Example
Base table with all attributes ... CREATE TABLE accounts (acc_no number(10) not null, acc_name varchar2(50) not null, ... 12500 Adams 12507 Blake 12666 King 12875 Smith

Base table with all attributes ... ... is extended with the virtual (derived) column CREATE TABLE accounts (acc_no number(10) not null, acc_name varchar2(50) not null, ... acc_branch number(2) generated always as (to_number(substr(to_char(acc_no),1,2)))‏ 12500 Adams 12 12507 Blake 12 12666 King 12 12875 Smith 12

Base table with all attributes ... ... is extended with the virtual (derived) column ... and the virtual column is used as partitioning key CREATE TABLE accounts (acc_no number(10) not null, acc_name varchar2(50) not null, ... acc_branch number(2) generated always as (to_number(substr(to_char(acc_no),1,2)))‏ partition by list (acc_branch) ... 12500 Adams 12 32320 Jones 32 12507 Blake 12 32407 Clark 32 ... 12666 King 12 32758 Hurd 32 12875 Smith 12 32980 Phillips 32

Interval Partitioning

Interval Partitioning
Partitioning is key-enabling functionality for managing large volumes of data one logical object for application transparency multiple physical segments for administration but Physical segmentation requires additional data management overhead new partitions must be created on-time for new data Mar Jan Feb CDRs SQL Application Automate the partition management

Interval Partitioning
extension to range partitioning full automation for equi-sized range partitions Partitions are created as metadata information only start partition is made persistent Segments are allocated as soon as new data arrives no need to create new partitions local indexes are created and maintained as well Interval “list of value” partitioning planned (e.g. list by geography) Interval “list by number” partitioning works now with range No need for any partition management

Interval Partitioning How it works
CREATE TABLE sales (order_date DATE, ...)‏ PARTITON BY RANGE (order_date)‏ INTERVAL(NUMTOYMINTERVAL(1,'month')‏ (PARTITION p_first VALUES LESS THAN ('01-FEB-2006'); Table SALES Jan 2006 ... Feb 2006 Mar 2006 Jan 2007 Oct 2009 Nov 2009 First segment is created NUMTOYMINTERVAL converts number n to an INTERVAL YEAR TO MONTH If multiple tablespaces defined, they are allocated in a round-robin fashion, think simplicity vs. flexibility

CREATE TABLE sales (order_date DATE, ...)‏ PARTITON BY RANGE (order_date)‏ INTERVAL(NUMTOYMINTERVAL(1,'month')‏ (PARTITION p_first VALUES LESS THAN ('01-FEB-2006'); Table SALES ... ... ... Jan 2006 Feb 2006 Mar 2006 Jan 2007 Oct 2009 Nov 2009 Other partitions only exist in metadata

CREATE TABLE sales (order_date DATE, ...)‏ PARTITON BY RANGE (order_date)‏ INTERVAL(NUMTOYMINTERVAL(1,'month')‏ (PARTITION p_first VALUES LESS THAN ('01-FEB-2006'); Table SALES ... ... ... Jan 2006 Feb 2006 Mar 2006 Jan 2007 Oct 2009 Nov 2009 New segment is automatically allocated INSERT INTO sales (order_date DATE, ...)‏ VALUES ('04-MAR-2006',...);

CREATE TABLE sales (order_date DATE, ...)‏ PARTITON BY RANGE (order_date)‏ INTERVAL(NUMTOYMINTERVAL(1,'month')‏ (PARTITION p_first VALUES LESS THAN ('01-FEB-2006'); Table SALES ... ... ... Jan 2006 Feb 2006 Mar 2006 Jan 2007 Oct 2009 Nov 2009 ... whenever data for a new partition arrives INSERT INTO sales (order_date DATE, ...)‏ VALUES ('17-OCT-2009',...);

Interval partitioned table can have classical range and automated interval section Automated new partition management plus full partition maintenance capabilities: “Best of both worlds” Table SALES ... ... ... Jan 2006 If data now arrives in a “hole”, e.g. Jan 2007, internal partitioning will go back and create a new partition Feb 2006 Mar 2006 Jan 2007 Oct 2009 Nov 2009

Interval partitioned table can have classical range and automated interval section Automated new partition management plus full partition maintenance capabilities: “Best of both worlds” Table SALES ... ... ... 2006 Jan 2007 Oct 2009 Nov 2009 Range partition section MERGE and move old partitions for ILM

Interval partitioned table can have classical range and automated interval section Automated new partition management plus full partition maintenance capabilities: “Best of both worlds” Table SALES ... ... ... you can change the interval of an interval partitioned table, existing intervals remain unaffected. 2006 Jan 2007 Oct 2009 Nov 2009 Range partition section Interval partition section MERGE and move old partitions for ILM Insert new data - Automatic segment creation INSERT INTO sales (order_date DATE, ...)‏ VALUES ('13-NOV-2009',...);

REF Partitioning

REF Partitioning Business Problem Solution
Related tables benefit from same partitioning strategy e.g. order – lineitem Redundant storage of the same information solves this problem data overhead maintenance overhead Solution Oracle Database 11g introduces REF Partitioning child table inherits the partitioning strategy of parent table through PK-FK relationship intuitive modelling Enhanced Performance and Manageability Predominantly useful for pruning and Partition Wise Joins 10.2 predicate had to be identical to partitioning strategy for PWJ 11.1 predicate can be different and PWJ still work. E.g. partitioning on order_date, search on order_id Used for nested table partitioning, and hence XML partitioning

Before REF Partitioning
Table ORDERS RANGE(order_date)‏ Primary key order_id ... ... Jan 2006 Feb 2006 Redundant storage of order_date Redundant maintenance Table LINEITEMS RANGE(order_date)‏ Foreign key order_id ... ... Jan 2006 Feb 2006

REF Partitioning ... ... ... ... Table ORDERS RANGE(order_date)‏
Primary key order_id ... ... Jan 2006 Feb 2006 PARTITION BY REFERENCE Partitioning key inherited through PK-FK relationship Table LINEITEMS Caveat: REF partitioning does not work with INTERVAL or VIRTUAL columns (maybe 11.2) RANGE(order_date)‏ Foreign key order_id ... ... Jan 2006 Feb 2006

Partitioning Advisor Packaged Apps Custom Apps SQL Workload Considers entire query workload to improve query performance Advises on partitioning methods Range (equal-interval), range key and interval Hash, hash key Integrated, non-conflicting advice with Indexes, MVs New! SQL Advisor SQL Plan Tuning SQL Structure Analysis Access Analysis Partition Analysis SQL Profile SQL Advice Indexes & MVs Partition Advice Caveat: partition advisor does not recommend changing / dropping existing partitions because it does not understand maintenance operations. Well-tuned SQL & Schema

Oracle Partitioning Summary
Proven functionality in 7th generation experience comes with age and customer usage Fundamental concepts for success the most comprehensive and complete offering 8th generation introduces ground-breaking new functionality extended COMPOSITE strategies further flexibility and performance INTERVAL for simplicity, ease of management VIRTUAL COLUMNS extend applicability with low maintenance REF enhanced performance for related tables Example of Customer Usage: Yahoo has 430,000 partitions in single table. Note: there is also a new feature called “system partitioning” which was implemented mainly for our domain index folks. It is a partitioning without any logical meaning. We create a partitioned table having x partitions; that's it. Everything else is up to the application, including data insertion which only works with the partition extended syntax.

A Q &

Advanced Compression Option

Challenges Explosion in data volume managed by Enterprises
Government regulations (Sarbanes-Oxley, HIPPA, etc) User generated content (Web 2.0) IT managers must support larger volumes of data with limited technology budgets Need to optimize storage consumption Also maintain acceptable application performance Intelligent and efficient compression technology can help address these challenges The volume of data being generated and retained by businesses has seen an exponential growth in the last few years. On an average, it is not uncommon to see data growing by 2-3 times every couple of years thanks to a variety of factors such as government regulations that mandate storing historical data for a certain number of years, massive amount of user generated content through web 2.0 applications and increasing adoption of multimedia content – enabled in turn by growing penetration of broadband technologies. Consequently, terabytes sized database – that were a novelty in the past – have becomes so commonplace these days. A key challenge before enterprises today is how to manage this enormous volume of data in a cost effective manner as they still have a limited IT budget. On a more technical level, there is also an associated challenge of ensuring the application performance does not degrade unacceptably while accessing this enormously large volume of data. This brings compression technology at forefront. We have known and used compression technology in some form of other so it should not need much explanation. The most prominent benefit of compression of course is the reduced storage consumption. However, the compression technology needs to be highly efficient and intelligent before it can be used with enterprise application in view of their demanding performance and scalability requirements.

Introducing Advanced Compression Option
Oracle Database 11g introduces a comprehensive set of compression capabilities Structured/Relational data compression Unstructured data compression Compression for backup data Network transport compression Reduces resource requirements and costs Storage System Network Bandwidth Memory Usage Oracle database 11g introduces a comprehensive set of technologies to address these challenges. Thanks to a number of innovative technology offering introduced in this release, our customer can compress practically any type of data – be it simple number and text traditionally stored in the database a.k.a structured data, or unstructured data such as documents, images, multimedia, etc or data being backed up. In addition, Oracle database 11g can also compress some of network traffic. Together these compression technology maximize the efficiency of resource utilization and reduces overall resource requirements and costs. Redo logs Standby Backups

Table Compression Introduced in Oracle9i Release 2
Supports compression during bulk load operations (Direct Load, CTAS) Data modified using conventional DML not compressed Optimized compression algorithm for relational data Improved performance for queries accessing large amounts of data Fewer IOs Buffer Cache efficiency Data is compressed at the database block level Each block contains own compression metadata – improves IO efficiency Local symbol table dynamically adapts to data changes Compression can be specified at either the table or partition levels Completely transparent to applications Noticeable impact on write performance Being self-contained is good b/c in order to read a compressed block we don’t have to access ‘other blocks’ that contain the symbol table or compression dictionary. For example, IBM has a database-wide symbol table. So every time you read a block that is compressed – you also have to read the blocks holding the db-wide symbol table – meaning extra I/O. We don’t have to do that!!

OLTP Table Compression
Oracle Database 11g extends compression for OLTP data Support for conventional DML Operations (INSERT, UPDATE, DELETE) New algorithm significantly reduces write overhead Batched compression ensures no impact for most OLTP transactions No impact on reads Reads may actually see improved performance due to fewer IOs and enhanced memory efficiency Lets first talk about compression of regular relational data. We were the early pioneer in terms of compression technology for databases when we introduced table compression for bulk load operations in Oracle9i. Using this feature, customers can compress data at the time of doing bulk load using operations such as DIRECTLOAD, CTAS, etc. However, until now, compression was not available for regular data manipulation operations such as INSERT, UPDATE and DELETES. In Oracle database 11g, we have extended the compression technology to support these operations as well. Consequently, compression in Oracle Database 11g can be used for all kinds of workload – be it OLTP or DW. It is important to mention here is that table compression enhancements introduced Oracle database 11g are not just incremental changes. Rather enormous amount of work has gone into making sure that the new compression technology has negligible impact on updates since any noticeable write time penalty due to compression won’t be acceptable in OLTP environment. As a result compression technology in Oracle database 11g is very efficient which could reduce the space consumption up to 2 to 3 times. And while you do that, not only your write performance does not degrade, your read performance or queries actually improves. This is because unlike desktop based compression techniques where you have to wait for data to uncompressed, our technology reads the compressed data directly and does not require any uncompression. This make sure that our query performance does not degrade because of compression. Now, the question is how does the query performance improve. It does so because we need to fetch less data to answer a query due to compression. Another advantage of fetching less data to satisfy a query is that we can now use the memory used to buffer the data read from disk more efficient thereby maximizing the memory utilization efficiency. Finally, just to make sure that I do not miss any thing obvious, our compression technology is completely application transparent. That mean that you can use this technology with any home grown or packaged application such as SAP, Siebel , EBS, etc.

OLTP Table Compression
Overhead Inserts are again uncompressed Free Space Uncompressed Compressed Block usage reaches PCTFREE – triggers Compression Inserts are uncompressed Block usage reaches PCTFREE – triggers Compression Adaptable, continuous compression Compression automatically triggered when block usage reaches PCTFREE Compression eliminates holes created due to deletions and maximizes contiguous free space in block

Using OLTP Table Compression
Requires database compatibility level at 11.1 or greater New Syntax extends the ‘COMPRESS’ keyword COMPRESS [FOR {ALL | DIRECT_LOAD} OPERATIONS] DIRECT_LOAD (DEFAULT) Refers to Bulk load operations from 10g and prior releases ALL OLTP + Direct loads Enable compression for a new table Enable only direct load compression on existing table only new rows are compressed, existing rows are uncompressed CREATE TABLE t1 COMPRESS FOR ALL OPERATIONS ALTER TABLE t2 COMPRESS

SecureFiles - Deduplication
Secure hash Enables storage of a single physical image for duplicate data Significantly reduces space consumption Dramatically improves write and copy operations No adverse impact on read operations may actually improve read performance for cache data Duplicate detection happens within a table, partition or sub-partition Specially useful for content management, applications and data archival applications

SecureFiles - Compression
Huge storage savings Industry standard compression algorithms 2-3x compression for typical files (doc, pdf, xml) Minimal CPU overhead during compression Automatically detects if SecureFile data is compressible Skips compression for already compressed data Auto-turn off compression when space savings are minimal or zero Two levels of compression provide different compression ratios Compression Levels: MEDIUM (default), HIGH Higher the degree of compression, higher the latency and CPU overhead incurred SecureFiles Compression is independent of table or index compression Server-side compression Allows for random reads and writes to SecureFile data Can be specified at a partition level Compression algorithm : ZLIB (support for more algorithms upcoming) Compression Overhead: 3- 5% Random reads/writes: utl_compress PL/SQL package which provide for compression/decompression functionality – does NOT provide for random access for read/write. With SecureFiles, only the required set of data blocks are uncompressed for providing random read/write access. More over, this is completely transparent to user applications. SecureFiles automatically maintains the mapping between uncompressed and compressed offsets. If the compression level is changed from MEDIUM to HIGH, there is no need to change the user application as the mapping is automatically updated to reflect the new algorithm.

Data Pump Compression Metadata compression available since Oracle Database 10g Oracle Database 11g extends compression to table data during exports no need to decompress before import Single step compression of both data and metadata compressed data directly hits disk resulting in reduced disk space requirements 75% reduction in dump file size on export of sample OE and SH schemas Compression factor comparable to GNU gzip utility Application transparent complete Data Pump functionality available on compressed files

Backup data and Network transport Compression
Fast RMAN Compression compresses the backup set contents before writing them to disk or tape no extra decompression steps are required during recovery when you use RMAN compression. high performance, industry standard compression algorithm 40% faster backup compression versus Oracle Database 10g suitable for fast, incremental daily backups reduces network usage Data Guard Network Compression compression of redo traffic over the network improves redo transport performance gap resolution is up to 2x faster RMAN compression: The option only includes the new zlib algorithm. The older bzip2 algorithm is still free. Zlib is faster than bzip2 but with a lesser compression ratio.

Summary Advanced Compression Option contains comprehensive data compression capabilities for all types of data Structured, Unstructured, Backup, Network Transport Reduces storage consumption by 2 to 3 times Improves read performance Enhances memory, buffer cache utilization Complete application transparency Benefits diverse application workloads So, to sum it up, as we have seen Advanced Compression Option in Oracle Database 11g contains a compelling set of compression technologies to help customers deal with data volume explosion in a cost effective manner without compromising application performance. Our compression technologies not only reduce disk space requirements, they improve application performance, enhance memory and network efficiency and can be used with any application out of the box. We are very excited about this technology and truly hope that this offering will help customers manage their large volume of data in the most cost effective manner.

VLDB Partitioning Compression

Bring up on stage two customers to tell the audience about their experiences.
Manpower Associates is a $14.9B global company with 27,000 employees in the temporary staffing business. Manpower runs a combined PeopleSoft Enterprise and JD Edwards EnterpriseOne shop. These experts in human resources use Enterprise HCM for their own staffing and EnterpriseOne Payroll and Service Billing for handling the large volumes of US-based temporary staff. Manpower is very happy with Oracle’s support since purchasing PeopleSoft and is looking forward to a long relationship with Oracle. Spokesperson will be Jay Schaudies, Vice President, Global eCommerce. Welch Foods is the food processing and marketing arm of National Grape Cooperative Association. Organized in 1945, National Grape is a grower-owned agricultural cooperative with 1,461 members. The company, headquartered in Concord, Massachusetts, operates six plants located in Michigan, New York, Pennsylvania and Washington. The company was running a mix of legacy, home grown, and manual systems that failed to provide senior management with accurate and timely cost and production information. Welch’s required a centralized manufacturing and financial information system to improve management decision making. The solution had to be hot-pluggable with existing technologies, for example, Welch’s Plumtree portal. Welch Foods chose Oracle over SAP for this business-critical application. The key to the customer’s business problem was their ability to manage costs. The company’s costs are driven by fruit solid content in each of their products, and they use a specialized technique called BRIX for measuring and calculating the cost of materials. Welch’s compared SAP and Oracle SAP’s software was too rigid and, therefore, unable to include the BRIX calculation in their manufacturing solution. Only Oracle’s OPM could bind this custom cost method into the Quality Management Process. Technology customer yet to be determined. Current possibilities include eBay and FTD Florists.

Oracle Partitioning in Oracle Database 11g

Similar presentations

Presentation on theme: "Oracle Partitioning in Oracle Database 11g"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Oracle Partitioning in Oracle Database 11g

Similar presentations

Presentation on theme: "Oracle Partitioning in Oracle Database 11g"— Presentation transcript:

Similar presentations

About project

Feedback