1 © 2008 Julian Dyke LOB Internals Julian Dyke Independent Consultant juliandyke.com Web Version - December 2008.

Slides:



Advertisements
Similar presentations
9 Creating and Managing Tables. Objectives After completing this lesson, you should be able to do the following: Describe the main database objects Create.
Advertisements

Data Definition Language (DDL)
The Architecture of Oracle
<Insert Picture Here>
Enqueue Waits : Locks. #.2 Copyright 2006 Kyle Hailey Wait Tree - Locks Waits Disk I/O Library Cache Enqueue Undo TX 6 Row Lock TX 4 ITL Lock HW Lock.
1 © 2006 Julian Dyke Supplemental Logging Julian Dyke Independent Consultant juliandyke.com Web Version.
Acknowledgments Byron Bush, Scott S. Hilpert and Lee, JeongKyu
Oracle Architecture. Instances and Databases (1/2)
Oracle 10g Database Administrator: Implementation and Administration Chapter 8 Advanced Table Management.
Basic Storage Concepts and Settings
1 - Oracle Server Architecture Overview
Harvard University Oracle Database Administration Session 2 System Level.
Harvard University Oracle Database Administration Session 5 Data Storage.
Chapter 5 Configuring the RMAN Environment. Objectives Show command to see existing settings Configure command to change settings Backing up the controlfile.
Working with SQL and PL/SQL/ Session 1 / 1 of 27 SQL Server Architecture.
1 juliandyke.com © 2005 Julian Dyke Data Segment Compression Julian Dyke Independent Consultant Web Version.
Oracle Database Administration Database files Logical database structures.
CHAPTER 11 Large Objects. Need for Large Objects Data type to store objects that contain large amount of text, log, image, video, or audio data. Most.
Oracle Database Architecture An Oracle server: –Is a database management system that provides an open, comprehensive, integrated approach to information.
Chapter Oracle Server An Oracle Server consists of an Oracle database (stored data, control and log files.) The Server will support SQL to define.
Case study DATABASE MANAGEMENT SYSTEMS Oracle Database 11g Release 2 (11.2) – MySQL 5.5 –
Database Technical Session By: Prof. Adarsh Patel.
1 Transaction Internals Julian Dyke Independent Consultant Web Version juliandyke.com © 2007 Julian Dyke.
By Lecturer / Aisha Dawood 1.  You can control the number of dispatcher processes in the instance. Unlike the number of shared servers, the number of.
7202ICT Database Administration Lecture 7 Managing Database Storage Part 2 Orale Concept Manuel Chapter 3 & 4.
Extents, segments and blocks in detail. Database structure Database Table spaces Segment Extent Oracle block O/S block Data file logical physical.
Chapter 6 1 © Prentice Hall, 2002 The Physical Design Stage of SDLC (figures 2.4, 2.5 revisited) Project Identification and Selection Project Initiation.
1 RAC Internals Julian Dyke Independent Consultant Web Version juliandyke.com © 2007 Julian Dyke.
Oracle Advanced Compression – Reduce Storage, Reduce Costs, Increase Performance Session: S Gregg Christman -- Senior Product Manager Vineet Marwah.
The protection of the DB against intentional or unintentional threats using computer-based or non- computer-based controls. Database Security – Part 2.
Oracle Database Administration
9 Storage Structure and Relationships. 9-2 Objectives Listing the different segment types and their uses Controlling the use of extents by segments Stating.
Oracle9i Database Administrator: Implementation and Administration 1 Chapter 7 Basic Table Management.
5/24/01 Leveraging SQL Server 2000 in ColdFusion Applications December 9, 2003 Chris Lomvardias SRA International
Sizing Basics  Why Size?  When to size  Sizing issues:  Bits and Bytes  Blocks (aka pages) of Data  Different Data types  Row Size  Table Sizing.
1 CS 430 Database Theory Winter 2005 Lecture 16: Inside a DBMS.
8-1 Copyright  Oracle Corporation, All rights reserved. Objects in Oracle8 1.Object Types - object-relational database 2.Large Objects (LOB) types.
Database structure and space Management. Database Structure An ORACLE database has both a physical and logical structure. By separating physical and logical.
Guide to Oracle 10g ITBIS373 Database Development Lecture 4a - Chapter 4: Using SQL Queries to Insert, Update, Delete, and View Data.
SQL Fundamentals  SQL: Structured Query Language is a simple and powerful language used to create, access, and manipulate data and structure in the database.
7 Copyright © 2005, Oracle. All rights reserved. Managing Undo Data.
Database structure and space Management. Segments The level of logical database storage above an extent is called a segment. A segment is a set of extents.
1 © 2010 Julian Dyke Edition-Based Redefinition Julian Dyke Independent Consultant juliandyke.com Web Version.
Dale Roberts Department of Computer and Information Science, School of Science, IUPUI Dale Roberts, Lecturer Computer Science, IUPUI
Week 3 Lecture 2 Basic Storage Concepts and Settings.
14 Copyright © 2006, Oracle. All rights reserved. Tuning Block Space Usage.
Week 4 Lecture 2 Advanced Table Management. Learning Objectives  Create tables with large object (LOB) columns and tables that are index-organized 
Harvard University Oracle Database Administration Session 6 Object Storage.
1 Logical I/O Julian Dyke Independent Consultant Web Version juliandyke.com © 2005 Julian Dyke.
9 Copyright © 2004, Oracle. All rights reserved. Manipulating Large Objects.
Oracle 10g Database Administrator: Implementation and Administration Chapter 5 Basic Storage Concepts and Settings.
Physical Database Design Purpose- translate the logical description of data into the technical specifications for storing and retrieving data Goal - create.
1 © 2005 Julian Dyke Reducing Redo Julian Dyke Independent Consultant Web Version juliandyke.com.
Chapter 4 Logical & Physical Database Design
Chapter 1Oracle9i DBA II: Backup/Recovery and Network Administration 1 Chapter 1 Backup and Recovery Overview MSCD642 Backup and Recovery.
It consists of two parts: collection of files – stores related data directory structure – organizes & provides information Some file systems may have.
Preface 1Performance Tuning Methodology: A Review Course Structure 1-2 Lesson Objective 1-3 Concepts 1-4 Determining the Worst Bottleneck 1-5 Understanding.
Academic Year 2015 Autumn. MODULE CC2006NI: Data Modelling and Database Systems Academic Year 2015 Autumn.
1 Part IV: Object-Relational Databases  Ch. 30: Types, Object Views, Methods  Ch. 31: Nested Tables and Varying Arrays  Ch. 32: Large Objects  Ch.
8 Copyright © 2007, Oracle. All rights reserved. Implementing SecureFile LOBs.
Select Operation Strategies And Indexing (Chapter 8)
Introduction To Oracle
Compression and Storage Optimization IDS xC4 Kevin Cherkauer
Database structure and space Management
Storage Structure and Relationships
Oracle 10g Database Administrator: Implementation and Administration
DATABASE MANAGEMENT SYSTEM
Transaction Internals
Object-Relational Features
Presentation transcript:

1 © 2008 Julian Dyke LOB Internals Julian Dyke Independent Consultant juliandyke.com Web Version - December 2008

© 2008 Julian Dyke juliandyke.com 2 Objectives 1. Understand how LOBs use storage in the Oracle database 2. Consider options to optimize LOB performance in terms of:  Reads  Writes  Impact on physical I/O  Impact on buffer cache

© 2008 Julian Dyke juliandyke.com 3 Agenda  Basic Files  Secure Files

© 2008 Julian Dyke juliandyke.com 4 Basic Files

© 2008 Julian Dyke juliandyke.com 5 Basic Files Overview  Introduced in Oracle 8.0  Known as Basic Files in Oracle 11.1 and above  Intended to replace LONG columns  Can be used to store large blocks of unstructured data e.g.:  text  graphics  video clips  sound waveforms  XML documents  Can store either character or binary data  Can be stored  within database (internal)  outside database (external)

© 2008 Julian Dyke juliandyke.com 6 Basic Files LOB Types  There are four types of LOB:  BLOB - Binary Large Object  Stored within database  Contains raw data  CLOB - Character Large Object  Stored within database  Supports database character set  NCLOB - NLS Character Large Object  Stored within database  Supports NLS character set  BFILE - Binary File Large Object  Stored in a binary file outside the database  Pointed to by a file locator within the database

© 2008 Julian Dyke juliandyke.com 7 Basic Files Internal LOBs  Stored outside of database  BLOB, CLOB or NCLOB  Maximum size is dependent on version and chunk size  Support features such as:  concurrency  redo logging and recovery  transactions with commits and rollbacks  For each row with an internal LOB column, the LOB can:  Be NULL  Point to an empty BLOB, CLOB or NCLOB  Point to a BLOB, CLOB or NCLOB value

© 2008 Julian Dyke juliandyke.com 8 Basic Files LOB columns versus LONG columns  LOBs are intended to replace LONG columns:  Oracle recommends converting LONG columns to LOB columns  LONG columns still used by many applications  LONG columns still used in Oracle 11.1 data dictionary tables including:  COL$, TABPART$, INDPART$, TRIGGER$, VIEW$ and CDEF$  Therefore it is unlikely LONG columns will be desupported in the near future  LONGs  Single column per table  Maximum size 2GB  Data always stored in-line  Cannot be an object attribute  Cannot be partitioned  Cannot be used in IOTs  Cannot be replicated  Access is sequential  Partial PL/SQL support  LOBs  Multiple columns per table  Maximum size 4GB or more  Data stored in-line or out-of-line  Can be an object attribute  Can be partitioned  Can be used in IOTs  Can be replicated  Access can be is random  Full PL/SQL support

© 2008 Julian Dyke juliandyke.com 9 Basic Files DBMS_LOB Package  BLOB and CLOB Subroutines  GETOPTIONS (11.1+)  GET_STORAGE_LIMIT  INSTR ISSECUREFILE  ISTEMPORARY  ISOPEN  LOADFROMFILE  LOADBLOBFROMFILE  LOADCLOBFROMFILE  OPEN  READ  SETOPTIONS (11.1+)  SUBSTR  TRIM  WRITE  WRITEAPPEND  APPEND  CLOSE  COMPARE  CONVERTTOBLOB  CONVERTTOCLOB  COPY  CREATETEMPORARY  ERASE  FRAGMENT_DELETE (11.1+)  FRAGMENT_INSERT (11.1+)  FRAGMENT_MOVE (11.1+)  FRAGMENT_REPLACE (11.1+)  FREETEMPORARY  GETCHUNKSIZE  GET_DEDUPLICATE_REGIONS (11.1+)  GETLENGTH

© 2008 Julian Dyke juliandyke.com 10 Basic Files DBMS_LOB Package  BFILE Subroutines  CLOSE  COMPARE  FILECLOSE  FILECLOSEALL  FILEEXISTS  FILEGETNAME  FILEISOPEN  FILEOPEN  GETLENGTH  INSTR  ISOPEN  OPEN  READ  SUBSTR

© 2008 Julian Dyke juliandyke.com 11 Basic Files DBA_LOBS View Column NameData TypeComments OWNERVARCHAR2(30) TABLE_NAMEVARCHAR2(30) COLUMN_NAMEVARCHAR2(4000) SEGMENT_NAMEVARCHAR2(30) TABLESPACE_NAMEVARCHAR2(30) INDEX_NAMEVARCHAR2(30) CHUNKNUMBER PCTVERSIONNUMBER RETENTIONNUMBER FREEPOOLSNUMBER CACHEVARCHAR2(10) LOGGINGVARCHAR2(7) ENCRYPTVARCHAR2(4)Oracle 11.1 and above COMPRESSIONVARCHAR2(6)Oracle 11.1 and above DEDUPLICATIONVARCHAR2(15)Oracle 11.1 and above IN_ROWVARCHAR2(3) FORMATVARCHAR2(15) PARTITIONEDVARCHAR2(3) SECUREFILEVARCHAR2(3)Oracle 11.1 and above

© 2008 Julian Dyke juliandyke.com 12 Basic Files LOB Objects  If a table includes an internal LOB column, the following objects will be created  LOB index  LOB data  Each object has a separate segment  For example: CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, ); LOB SYS_IL C00002$$ LOB INDEX LOB SYS_LOB C00002$$ LOB DATA LOB T1 TABLE

© 2008 Julian Dyke juliandyke.com 13 Basic Files LOB Segments  Every LOB has a LOB data segment and a LOB index segment  LOB data segment is reported by DBA_OBJECTS  Segment name is system generated e.g. SYS_LOB C00002$$  where:  is the OBJECT_ID of the table  is the column number of the LOB column within the table  LOB index is not reported by DBA_OBJECTS  Index is still defined in SYS.OBJ$ and SYS.IND$  Index name is reported by DBA_LOBS e.g. SYS_IL C00002$$  where:  is the OBJECT_ID of the table  is the column number of the LOB column within the table

© 2008 Julian Dyke juliandyke.com 14 Basic Files LOB Objects  One LOB index segment and one LOB data segment will be created for each LOB column  For example: CREATE TABLE t2 ( c1 NUMBER, c2 CLOB, c3 CLOB ); LOB SYS_IL C00002$$ LOB INDEX LOB SYS_IL C00003$$ LOB INDEX LOB SYS_LOB C00003$$ LOB DATA LOB SYS_LOB C00002$$ LOB DATA LOB T2 TABLE

© 2008 Julian Dyke juliandyke.com 15 Basic Files Partitioned LOBs  Oracle supports partitioned and sub-partitioned LOBS in Oracle and above  LOBs are supported in partitioned IOTs in Oracle and above  LOB and and LOB index segments are equi-partitioned with base table  Partitioning key cannot contain a LOB column CREATE TABLE t3 ( c1 NUMBER, c2 CLOB ) PARTITION BY RANGE (c1) ( PARTITION p1 VALUES LESS THAN (100), PARTITION p2 VALUES LESS THAN (200) );

© 2008 Julian Dyke juliandyke.com 16 Basic Files Partitioned LOBs SYS_LOB_P22 LOB PARTITION SYS_LOB_P21 LOB PARTITION LOB SYS_LOB C00002$$ LOB SYS_LOB_P24 INDEX PARTITION SYS_LOB_P23 INDEX PARTITION LOB SYS_IL C00002$$ LOB INDEX P2 TABLE PARTITION P1 TABLE PARTITION LOB T3 TABLE

© 2008 Julian Dyke juliandyke.com 17 Basic Files Sub Partitioned LOBs CREATE TABLE t4 ( c1 NUMBER, c2 NUMBER, c3 CLOB ) PARTITION BY RANGE (c1) ( PARTITION p1 VALUES LESS THAN (100) ( SUBPARTITION p1s1 VALUES LESS THAN (10), SUBPARTITION p1s2 VALUES LESS THAN (20) ), PARTITION p2 VALUES LESS THAN (200) ( SUBPARTITION p2s1 VALUES LESS THAN (10), SUBPARTITION p2s2 VALUES LESS THAN (20) ) );

© 2008 Julian Dyke juliandyke.com 18 Basic Files Sub Partitioned LOBs SYS_IL_P23SYS_IL_P24 SYS_LOB_SUBP32 LOB SUBPARTITION SYS_LOB_SUBP33 LOB SUBPARTITION SYS_LOB_SUBP35 LOB SUBPARTITION SYS_LOB_SUBP36 LOB SUBPARTITION SYS_LOB_P34 LOB PARTITION SYS_LOB_P31 LOB PARTITION LOB SYS_LOB C00003$$ LOB SYS_LOB_SUBP32 INDEX SUBPARTITION SYS_LOB_SUBP33 INDEX SUBPARTITION SYS_LOB_SUBP35 INDEX SUBPARTITION SYS_LOB_SUBP36 INDEX SUBPARTITION SYS_LOB_P44 INDEX PARTITION SYS_LOB_P41 INDEX PARTITION LOB SYS_IL C00003$$ LOB INDEX P1S1 TABLE SUBPARTITION P1S2 TABLE SUBPARTITION P2 TABLE PARTITION P1 TABLE PARTITION LOB T4 TABLE P2S1 TABLE SUBPARTITION P2S2 TABLE SUBPARTITION

© 2008 Julian Dyke juliandyke.com 19 Basic Files In Row versus Out-of-Line Storage  Values can be stored  in-line - in a row piece  out-of-line - in a separate LOB segment  ENABLE STORAGE IN ROW (default)  LOB value is stored within row if size is < 3964 bytes  LOB value is stored out-of-line if size > 3964 bytes  If in-line LOB grows to more than 3964 bytes  LOB value is moved out-of-line to LOB segment  If out-of-line LOB shrinks to less than 3964 bytes  LOB value remains out-of-line in LOB segment  DISABLE STORAGE IN ROW  LOB value is always stored out-of-line in LOB segment CREATE TABLE t1 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (ENABLE STORAGE IN ROW); CREATE TABLE t2 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (DISABLE STORAGE IN ROW);

© 2008 Julian Dyke juliandyke.com 20 Basic Files LOB Initialization  LOBs are initialized using built-in functions  EMPTY_BLOB ()  EMPTY_CLOB ()  Both functions create LOB locators  Minimum size is 36 bytes  Maximum size is 86 bytes CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, ); INSERT INTO t1 VALUES (c1,NULL); INSERT INTO t1 VALUES (c1,EMPTY_CLOB()); C2 occupies 0 bytes C2 occupies 37bytes

© 2008 Julian Dyke juliandyke.com 21 Basic Files LOB Initialization  LOB locator is initialized using EMPTY_CLOB() or EMPTY_BLOB()  For example: c d tab 0, row tl: 44 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 3] c col 1: [36] c d INSERT INTO t1 VALUES (c1,EMPTY_CLOB());  Block dump includes : LOBIDHeader Body Length LOB LengthVersionFlags

© 2008 Julian Dyke juliandyke.com 22 Basic Files LOB ID  LOB ID is a 10 byte number identifying individual instance of a LOB  Allocated when LOB value is created including EMPTY_CLOB() etc  Format is where  is a currently unknown 4-byte number (always 1)  is a 6-byte number generated from sequence SYS.IDGEN$  For example: INSERT INTO t1 VALUES (0,EMPTY_CLOB) SELECT sequence_owner, sequence_name, nextvalue, increment_by, cache_size FROM v$_sequences; Sequence OwnerSequence NameNext ValueIncrement ByCache Size SYSIDGEN1$ SELECT sequence_owner, sequence_name, nextvalue, increment_by, cache_size FROM v$_sequences; Sequence OwnerSequence NameNext ValueIncrement ByCache Size SYSIDGEN1$  LOB ID =(0x9219 = 37401)

© 2008 Julian Dyke juliandyke.com 23 Basic Files LOB Inline Storage  Example byte LOB - ENABLE STORAGE IN ROW c a7b 001c0900 0c ab 0, row tl: 54 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [48] c a 7b 00 1c c INSERT INTO t1 VALUES (c1, LPAD ('X',12,'X')); 0x0C = 12 0x1C = 28

© 2008 Julian Dyke juliandyke.com 24 Basic Files LOB Inline Storage  Example byte LOB - ENABLE STORAGE IN ROW c aad 01a tab 0, row tl: 444 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [436] c a ad 01 a INSERT INTO t1 VALUES (c1,LPAD ('X',400,'X')); 0x0190 = 400 0x01A0 = 416

© 2008 Julian Dyke juliandyke.com 25 Basic Files LOB Inline Storage  Example byte LOB - ENABLE STORAGE IN ROW c ca1 0f8c0900 0f7c tab 0, row tl: 4008 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [4000] c c a1 0f 8c f 7c INSERT INTO t1 VALUES (c1,LPAD ('X',3964,'X')); 0x0F7C = x0F8C = 3980

© 2008 Julian Dyke juliandyke.com 26 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c d f7d b5 tab 0, row tl: 46 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [40] c d f 7d b5 INSERT INTO t1 VALUES (c1,LPAD ('X',3965,'X')); 0x0014 = 200x0F7D bytes Address of first chunk 0x0F7D = 3965

© 2008 Julian Dyke juliandyke.com 27 Basic Files LOB Out-of-Line Storage  Out-of-line LOB data is stored in a PAGETABLE MANAGED LOB BLOCK  Each block has  a 56 byte header  a 4 byte trailer  For example for a 8192 byte block Trailer 4 bytes Header 56 bytes LOB Data 8132 bytes

© 2008 Julian Dyke juliandyke.com 28 CREATE DIRECTORY dir1 AS '/tmp'; CREATE TABLE t11 ( c1 NUMBER, c2 CLOB ); INSERT INTO t11 VALUES (0,EMPTY_CLOB()); Basic Files Out-of-Line Storage DECLARE l_bfile BFILE; l_clob CLOB; l_result NUMBER; BEGIN l_bfile := BFILENAME ('DIR1','dbms_stats.lst'); DBMS_LOB.OPEN (l_bfile); SELECT c2 INTO l_clob FROM t11 WHERE c1 = 0 FOR UPDATE; l_result := DBMS_LOB.GETLENGTH (l_bfile); DBMS_LOB.LOADFROMFILE (l_clob,l_bfile,l_result); DBMS_LOB.CLOSE (l_bfile); END;

© 2008 Julian Dyke juliandyke.com 29 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c dff f c d e e f a tl: 78 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [72] c d ff f c d e e f a INSERT INTO t1 VALUES (c1,BFILE...); 8 x 8132 = x1055 = chunks 0x34 = 52 DBA of LOB chunks 0x1055 bytes

© 2008 Julian Dyke juliandyke.com 30 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c c d e f c tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c c d e f c LOB Index exists but is empty 12 chunks 0 bytes 0x40 = 64 INSERT INTO t1 VALUES (c1,BFILE...); 12 x 8132 = 97584

© 2008 Julian Dyke juliandyke.com 31 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c c d e f c tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c c d e f c INSERT INTO t1 VALUES (c1,BFILE...); 12 chunks 1 byte 0x40 = 64 LOB Index is used for 13 th chunk

© 2008 Julian Dyke juliandyke.com 32 Basic Files LOB Out-of-Line Storage  Example 7 continued byte LOB - ENABLE STORAGE IN ROW  LOB Index f00 row#0[7982] flag: , lock: 2, len=50, data:(32): f col 0; len 10; (10): col 1; len 4; (4): c c Key Data First chunk number  LOB Index is always unique  Data is always 32 bytes LOBID

© 2008 Julian Dyke juliandyke.com 33 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c ed e f a b c d e tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c ed e f a b c d e INSERT INTO t1 VALUES (c1,BFILE...); 0x17 = 23 chunks 12 chunks in row 11 chunks in index 0xe4 = 228 bytes

© 2008 Julian Dyke juliandyke.com 34 Basic Files LOB Out-of-Line Storage  Example 8 continued byte LOB - ENABLE STORAGE IN ROW  Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): f a b c d e col 0; len 10; (10): ed col 1; len 4; (4): c row#1[7932] flag: , lock: 2, len=50, data:(32): col 0; len 10; (10): ed col 1; len 4; (4): f a b c d e ed00 0c ed00 14 Key Data

© 2008 Julian Dyke juliandyke.com 35 Basic Files Chunk Size  Each internal LOB has a chunk size  Chunks can be manipulated individually  Default chunk size is block size  Maximum chunk size is for all tablespace block sizes  Chunk size is:  specified in bytes  rounded up to nearest block size CREATE TABLE t3 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CHUNK 32768);  Chunk size is reported in DBA_LOB.CHUNK  Chunk size is stored in SYS.LOB$.CHUNK  Although chunk size is specified and reported in bytes, it is stored in LOB$ in terms of tablespace blocks

© 2008 Julian Dyke juliandyke.com 36 Basic Files Chunk Size  Example byte LOB  CHUNK = 8192 (default) c dff f c d e e f a c d5 001c d tab 0, row tl: 54 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [48] c d5 00 1c d  CHUNK = (maximum) 4 x 8132 = x = x1055 =

© 2008 Julian Dyke juliandyke.com 37 Basic Files Chunk Size  In Oracle 11.1 maximum LOB size is determined by chunk size: CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, c3 CLOB, c4 CLOB ) LOB (c2) STORE AS (CHUNK 8192), LOB (c3) STORE AS (CHUNK 16384), LOB (c4) STORE AS (CHUNK 32768); SELECT dbms_lob.getchunksize (c2), dbms_lob.get_storage_limit (c2), dbms_lob.getchunksize (c3), dbms_lob.get_storage_limit (c3), dbms_lob.getchunksize (c4), dbms_lob.get_storage_limit (c4) FROM t1; Chunk SizeStorage Limit# Chunks ,926,674,042,9404,294,967, ,853,348,085,8804,294,967, ,706,696,171,7604,294,967,295

© 2008 Julian Dyke juliandyke.com 38 Basic Files Read Consistency  PCTVERSION clause  Specifies the amount of space that will be used for the LOB consistent- read mechanism  Affects reclamation of old copies of LOB data  Affects the ability to perform consistent read  Range is 0 to 100  Default value is 10 CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (PCTVERSION 20);  Reported in DBA_LOB.PCTVERSION  Can be set to 0 for read-only LOBs  PCTVERSION should be set to a low value if:  updates and reads are not concurrent  the LOB is written once and then read-only  PCTVERSION should be set to a high value if:  there are large numbers of queries  there is heavy read and write activity

© 2008 Julian Dyke juliandyke.com 39 Basic Files Read Consistency  Example byte LOB - update 4000 bytes starting at offset  PCTVERSION = 10 (default) DECLARE l_clob CLOB; l_str VARCHAR2(4000); BEGIN l_str := LPAD ('Y',4000,'Y'); SELECT c2 INTO l_clob FROM t19 WHERE c1 = 0; FOR UPDATE; DBMS_LOB.WRITE (l_clob,4000,10000,l_str); END; /

© 2008 Julian Dyke juliandyke.com 40 Basic Files Read Consistency  Example 9 (continued)  Before Update tl: 62 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [56] c f fd f ff fc c f fd f010003ff fc c f fd fe010003ff fc  After Update Updated Chunk

© 2008 Julian Dyke juliandyke.com 41 Basic Files Read Consistency  Example 9 (continued) row#0[7982] flag: , lock: 2, len=50, data:(32): fe col 0; len 10; (10): f1 b col 1; len 4; (4): fe fe00 LOB Index LOB Data fec f Key Data Free List 3FD3FE3FF4003FC 3FE 40F STOP

© 2008 Julian Dyke juliandyke.com 42 Basic Files Read Consistency  Example byte LOB - update 4000 bytes starting at offset  PCTVERSION = 10 DECLARE l_clob CLOB; l_str VARCHAR2(4000); BEGIN l_str := LPAD ('Y',4000,'Y'); SELECT c2 INTO l_clob FROM t19 WHERE c1 = 0; FOR UPDATE; DBMS_LOB.WRITE (l_clob,4000,30000,l_str); END; /

© 2008 Julian Dyke juliandyke.com 43 Basic Files Read Consistency  Example 10 (continued)  Before Update tl: 62 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [56] c bf fd f ff fe c bf fd f010003ff010003fe c f fd f010003ff fc  After Update Updated Chunk

© 2008 Julian Dyke juliandyke.com 44 Basic Files Read Consistency  Example 10 (continued) - Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): fc col 0; len 10; (10): fe c col 1; len 4; (4): fc fc fec f Key Data LOB Index LOB Data Free List 3FD3FE3FF400 3FC3FE40F4003FE3FC410 STOP

© 2008 Julian Dyke juliandyke.com 45 Basic Files Partial Deletion  Example byte LOB - Erase bytes from offset DECLARE l_clob CLOB; l_amount NUMBER := 18000; BEGIN SELECT c2 INTO l_clob FROM t22 WHERE c1 = 0 FOR UPDATE; DBMS_LOB.ERASE (l_clob,l_amount,12000); END;

© 2008 Julian Dyke juliandyke.com 46 Basic Files Partial Deletion  Example 11 continued byte LOB - Erase bytes from offset  Before ERASE c b d d e c f a b tab 0, row tl: 78 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [72] c b d d e c f a b c b d e f c f a b  After ERASE

© 2008 Julian Dyke juliandyke.com 47 Basic Files Partial Deletion  Example 11 continued byte LOB - Erase bytes from offset  After ERASE - Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): e f col 0; len 10; (10): a col 1; len 4; (4): e e f LOB Index LOB Data Free List Data 46D 46E47046C47F48047A46E46F470 47B46F47D47E STOP

© 2008 Julian Dyke juliandyke.com 48 Basic Files CACHE Clause  Specifies whether a LOB will be stored in the database buffer cache when values are read and written.  Allows LOB values to be shared amongst users.  Possible values are CACHE, CACHE READS and NOCACHE  Default value is NOCACHE CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE);  Reported in DBA_LOB.CACHE  Can be NO, CACHEREADS or YES CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS);

© 2008 Julian Dyke juliandyke.com 49 Basic Files CACHE versus NOCACHE  CACHE LOGGING Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time Wait Class enq: CF - contention 6, Other CPU time log file sync 69, Commit control file parallel write 31, System I/O log file parallel write 60, System I/O Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time Wait Class CPU time log file sync 79, Commit log file parallel write 66, System I/O SQL*Net more data from client 37, Network db file parallel write 5, System I/O  NOCACHE NOLOGGING

© 2008 Julian Dyke juliandyke.com 50 Basic Files LOGGING  Possible values are LOGGING (default) and NOLOGGING  If CACHE option is specified then LOGGING is mandatory  LOGGING/NOLOGGING cannot be specified without CACHE clause CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE NOLOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS NOLOGGING);  Reported in DBA_LOB.LOGGING  Can be NO or YES

© 2008 Julian Dyke juliandyke.com 51 Basic Files LOB Compression  LOBs can be compressed using the LZ_COMPRESS procedure in the UTL_COMPRESS package  A quality can be specified  1 - Fastest  9 - Slowest  The default quality is 6  Compression achieved is not very consistent File size in bytes Qualitydbms_stats.lstalert.log Uncompressed

© 2008 Julian Dyke juliandyke.com 52 Basic Files LOB Compression  Example DECLARE l_bfile BFILE; l_blob BLOB; l_clob CLOB; l_length NUMBER; l_quality NUMBER := 1; l_dest_offset NUMBER := 1; l_source_offset NUMBER := 1; l_blob_csid NUMBER := DBMS_LOB.DEFAULT_CSID; l_lang_context NUMBER := DBMS_LOB.DEFAULT_LANG_CTX; l_warning NUMBER; BEGIN l_bfile := BFILENAME ('DIR1','alert.log'); DBMS_LOB.OPEN (l_bfile); DBMS_LOB.CREATETEMPORARY (l_blob,FALSE); l_length := DBMS_LOB.GETLENGTH (l_bfile); UTL_COMPRESS.LZ_COMPRESS (l_bfile,l_blob,l_quality); SELECT c2 INTO l_clob FROM t34 WHERE c1 = 0 FOR UPDATE; DBMS_LOB.CONVERTTOCLOB (l_clob,l_blob,DBMS_LOB.LOBMAXSIZE, l_dest_offset,l_source_offset,l_blob_csid,l_lang_context,l_warning); END; CREATE DIRECTORY dir1 AS '/tmp'; CREATE TABLE t34 (c1 NUMBER, c2 CLOB); INSERT INTO t34 VALUES (0,EMPTY_CLOB ());

© 2008 Julian Dyke juliandyke.com 53 Secure Files

© 2008 Julian Dyke juliandyke.com 54 Secure Files Overview  In Oracle 11.1 and above there are two types of LOB  STORE AS BASICFILE (default)  STORE AS SECUREFILE  STORE AS BASICFILE  Default  Behaves as Oracle 10.2 and below  STORE AS SECUREFILE  Oracle 11.1 and above  Supports several new features including  LOB compression  LOB encryption  LOB deduplication  Metadata only logging  Configurable retention time

© 2008 Julian Dyke juliandyke.com 55 SecureFiles Storage  Can only be created in an ASSM tablespace  New block types supporting SecureFiles include: DescriptionBlock Type NGLOB: Extent Map0x3c NGLOB: Hash Bucket0x3d NGLOB: Committed Free Space0x3e NGLOB: Segment Header0x3f NGLOB: Lob Extent Header0x40 NGLOB: Persistent Undo0x41  LOB Data Segments are still stored in blocks of type trans_data (block type 6)  Note  CHUNK is advisory - only maintained for backward compatibility  PCTVERSION not supported - use RETENTION instead

© 2008 Julian Dyke juliandyke.com 56 Secure File Initialization  EMPTY_CLOB ()  Basic File c a tl: 36 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [30] c a c d  Secure File

© 2008 Julian Dyke juliandyke.com 57 Secure File LOB Locator  Example DBMS_STATS.LST bytes  Basic File c b 001f e d tl: 57 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [51] c b 00 1f e d c dff f c d e e f a  Secure File DBA of first block in extent # Blocks in extent 0x10E75 = 69237

© 2008 Julian Dyke juliandyke.com 58 Secure Files Lob Blocks  Out-of-line LOB data is stored in a trans data block (block type 6)  Each block has  a 128 byte header  a 4 byte trailer  For example for a 8192 byte block Trailer 4 bytes Header 128 bytes LOB Data 8060 bytes

© 2008 Julian Dyke juliandyke.com 59 Secure Files LOB Compression  Oracle 11.1 and above  Enables server-side LOB compression  Random read/write access is still possible  Independent of table compression / index compression  Only valid for SecureFile LOBs  Valid values are  COMPRESS  COMPRESS MEDIUM  COMPRESS HIGH  NOCOMPRESS  COMPRESS MEDIUM  Faster (less resource)  COMPRESS HIGH  Better compression (less storage)  Individual LOBs can be compressed using DBMS_LOB.SETOPTIONS ()

© 2008 Julian Dyke juliandyke.com 60 Secure Files LOB Compression  Compressed size of LOB is not reported in Oracle 11.1  To calculate compressed size count number of extents  In block dump  Using DBMS_LOBUTIL.GETINODE  Size will be an (over) estimate  Example - alert.log - sizes in bytes Uncompressed Size COMPRESS MEDIUM72540 COMPRESS HIGH40300  SecureFile LOB Compression much better than UTLCOMPRESS package  For example best compression achieved for same file using LZ_COMPRESS was bytes

© 2008 Julian Dyke juliandyke.com 61 Secure Files DBMS_LOBUTIL  Unsupported package that describes SecureFile LOBs  Defined in $ORACLE_HOME/rdbms/admin/dbmslobu.sql  Subroutines include  GETINODE  GETLOBMAP  GETEXTENTS  GETINODE  Reports LOB header  GETLOBMAP  Reports LOB extent header  Numeric parameter is extent number (0..N-1)  GETEXTENTS  Reports LOB extents in more detail  Pipelined function - takes REF CURSOR as parameter

© 2008 Julian Dyke juliandyke.com 62 Secure Files DBMS_LOBUTIL DECLARE l_clob CLOB; l_inode DBMS_LOBUTIL_INODE_T; l_lobmap DBMS_LOBUTIL_LOBMAP_T; BEGIN SELECT c2 INTO l_clob FROM t29 WHERE c1 = 0 FOR UPDATE; l_inode := DBMS_LOBUTIL.GETINODE (l_clob); DBMS_OUTPUT.PUT_LINE ('LOBID '||RAWTOHEX (l_inode.lobid)); DBMS_OUTPUT.PUT_LINE (' flags '||l_inode.flags); DBMS_OUTPUT.PUT_LINE (' length '||l_inode.length); DBMS_OUTPUT.PUT_LINE (' version '||l_inode.version); DBMS_OUTPUT.PUT_LINE (' extents '||l_inode.extents); FOR i IN 0..l_inode.extents - 1 LOOP l_res := DBMS_LOBUTIL.GETLOBMAP (l_clob,i); DBMS_OUTPUT.PUT_LINE ('Extent# '||i); DBMS_OUTPUT.PUT_LINE (' rdba '||l_res.rdba); DBMS_OUTPUT.PUT_LINE (' nblks '||l_res.nblks); DBMS_OUTPUT.PUT_LINE (' offset '||l_res.offset); DBMS_OUTPUT.PUT_LINE (' length '||l_res.length); END LOOP; END;

© 2008 Julian Dyke juliandyke.com 63 References  Oracle Database Application Developer's Guide - Large Objects 10g Release 2 (10.2) - Part Number B  Oracle Database SecureFiles and Large Objects Developers Guide 11g Release 1 (11.1) - Part Number B

© 2008 Julian Dyke juliandyke.com 64 Any Questions ?