1 © 2008 Julian Dyke LOB Internals Julian Dyke Independent Consultant juliandyke.com Web Version - December 2008

© 2008 Julian Dyke juliandyke.com 2 Objectives 1. Understand how LOBs use storage in the Oracle database 2. Consider options to optimize LOB performance in terms of:  Reads  Writes  Impact on physical I/O  Impact on buffer cache

© 2008 Julian Dyke juliandyke.com 3 Agenda  Basic Files  Secure Files

© 2008 Julian Dyke juliandyke.com 4 Basic Files

© 2008 Julian Dyke juliandyke.com 5 Basic Files Overview  Introduced in Oracle 8.0  Known as Basic Files in Oracle 11.1 and above  Intended to replace LONG columns  Can be used to store large blocks of unstructured data e.g.:  text  graphics  video clips  sound waveforms  XML documents  Can store either character or binary data  Can be stored  within database (internal)  outside database (external)

© 2008 Julian Dyke juliandyke.com 6 Basic Files LOB Types  There are four types of LOB:  BLOB - Binary Large Object  Stored within database  Contains raw data  CLOB - Character Large Object  Stored within database  Supports database character set  NCLOB - NLS Character Large Object  Stored within database  Supports NLS character set  BFILE - Binary File Large Object  Stored in a binary file outside the database  Pointed to by a file locator within the database

© 2008 Julian Dyke juliandyke.com 7 Basic Files Internal LOBs  Stored outside of database  BLOB, CLOB or NCLOB  Maximum size is dependent on version and chunk size  Support features such as:  concurrency  redo logging and recovery  transactions with commits and rollbacks  For each row with an internal LOB column, the LOB can:  Be NULL  Point to an empty BLOB, CLOB or NCLOB  Point to a BLOB, CLOB or NCLOB value

© 2008 Julian Dyke juliandyke.com 8 Basic Files LOB columns versus LONG columns  LOBs are intended to replace LONG columns:  Oracle recommends converting LONG columns to LOB columns  LONG columns still used by many applications  LONG columns still used in Oracle 11.1 data dictionary tables including:  COL$, TABPART$, INDPART$, TRIGGER$, VIEW$ and CDEF$  Therefore it is unlikely LONG columns will be desupported in the near future  LONGs  Single column per table  Maximum size 2GB  Data always stored in-line  Cannot be an object attribute  Cannot be partitioned  Cannot be used in IOTs  Cannot be replicated  Access is sequential  Partial PL/SQL support  LOBs  Multiple columns per table  Maximum size 4GB or more  Data stored in-line or out-of-line  Can be an object attribute  Can be partitioned  Can be used in IOTs  Can be replicated  Access can be is random  Full PL/SQL support

© 2008 Julian Dyke juliandyke.com 9 Basic Files DBMS_LOB Package  BLOB and CLOB Subroutines  GETOPTIONS (11.1+)  GET_STORAGE_LIMIT  INSTR ISSECUREFILE  ISTEMPORARY  ISOPEN  LOADFROMFILE  LOADBLOBFROMFILE  LOADCLOBFROMFILE  OPEN  READ  SETOPTIONS (11.1+)  SUBSTR  TRIM  WRITE  WRITEAPPEND  APPEND  CLOSE  COMPARE  CONVERTTOBLOB  CONVERTTOCLOB  COPY  CREATETEMPORARY  ERASE  FRAGMENT_DELETE (11.1+)  FRAGMENT_INSERT (11.1+)  FRAGMENT_MOVE (11.1+)  FRAGMENT_REPLACE (11.1+)  FREETEMPORARY  GETCHUNKSIZE  GET_DEDUPLICATE_REGIONS (11.1+)  GETLENGTH

© 2008 Julian Dyke juliandyke.com 10 Basic Files DBMS_LOB Package  BFILE Subroutines  CLOSE  COMPARE  FILECLOSE  FILECLOSEALL  FILEEXISTS  FILEGETNAME  FILEISOPEN  FILEOPEN  GETLENGTH  INSTR  ISOPEN  OPEN  READ  SUBSTR

© 2008 Julian Dyke juliandyke.com 11 Basic Files DBA_LOBS View Column NameData TypeComments OWNERVARCHAR2(30) TABLE_NAMEVARCHAR2(30) COLUMN_NAMEVARCHAR2(4000) SEGMENT_NAMEVARCHAR2(30) TABLESPACE_NAMEVARCHAR2(30) INDEX_NAMEVARCHAR2(30) CHUNKNUMBER PCTVERSIONNUMBER RETENTIONNUMBER FREEPOOLSNUMBER CACHEVARCHAR2(10) LOGGINGVARCHAR2(7) ENCRYPTVARCHAR2(4)Oracle 11.1 and above COMPRESSIONVARCHAR2(6)Oracle 11.1 and above DEDUPLICATIONVARCHAR2(15)Oracle 11.1 and above IN_ROWVARCHAR2(3) FORMATVARCHAR2(15) PARTITIONEDVARCHAR2(3) SECUREFILEVARCHAR2(3)Oracle 11.1 and above

© 2008 Julian Dyke juliandyke.com 12 Basic Files LOB Objects  If a table includes an internal LOB column, the following objects will be created  LOB index  LOB data  Each object has a separate segment  For example: CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, ); LOB SYS_IL C00002$$ LOB INDEX LOB SYS_LOB C00002$$ LOB DATA LOB T1 TABLE

© 2008 Julian Dyke juliandyke.com 13 Basic Files LOB Segments  Every LOB has a LOB data segment and a LOB index segment  LOB data segment is reported by DBA_OBJECTS  Segment name is system generated e.g. SYS_LOB C00002$$  where:  is the OBJECT_ID of the table  is the column number of the LOB column within the table  LOB index is not reported by DBA_OBJECTS  Index is still defined in SYS.OBJ$ and SYS.IND$  Index name is reported by DBA_LOBS e.g. SYS_IL C00002$$  where:  is the OBJECT_ID of the table  is the column number of the LOB column within the table

© 2008 Julian Dyke juliandyke.com 14 Basic Files LOB Objects  One LOB index segment and one LOB data segment will be created for each LOB column  For example: CREATE TABLE t2 ( c1 NUMBER, c2 CLOB, c3 CLOB ); LOB SYS_IL C00002$$ LOB INDEX LOB SYS_IL C00003$$ LOB INDEX LOB SYS_LOB C00003$$ LOB DATA LOB SYS_LOB C00002$$ LOB DATA LOB T2 TABLE

© 2008 Julian Dyke juliandyke.com 15 Basic Files Partitioned LOBs  Oracle supports partitioned and sub-partitioned LOBS in Oracle and above  LOBs are supported in partitioned IOTs in Oracle and above  LOB and and LOB index segments are equi-partitioned with base table  Partitioning key cannot contain a LOB column CREATE TABLE t3 ( c1 NUMBER, c2 CLOB ) PARTITION BY RANGE (c1) ( PARTITION p1 VALUES LESS THAN (100), PARTITION p2 VALUES LESS THAN (200) );

© 2008 Julian Dyke juliandyke.com 16 Basic Files Partitioned LOBs SYS_LOB_P22 LOB PARTITION SYS_LOB_P21 LOB PARTITION LOB SYS_LOB C00002$$ LOB SYS_LOB_P24 INDEX PARTITION SYS_LOB_P23 INDEX PARTITION LOB SYS_IL C00002$$ LOB INDEX P2 TABLE PARTITION P1 TABLE PARTITION LOB T3 TABLE

© 2008 Julian Dyke juliandyke.com 17 Basic Files Sub Partitioned LOBs CREATE TABLE t4 ( c1 NUMBER, c2 NUMBER, c3 CLOB ) PARTITION BY RANGE (c1) ( PARTITION p1 VALUES LESS THAN (100) ( SUBPARTITION p1s1 VALUES LESS THAN (10), SUBPARTITION p1s2 VALUES LESS THAN (20) ), PARTITION p2 VALUES LESS THAN (200) ( SUBPARTITION p2s1 VALUES LESS THAN (10), SUBPARTITION p2s2 VALUES LESS THAN (20) ) );

© 2008 Julian Dyke juliandyke.com 18 Basic Files Sub Partitioned LOBs SYS_IL_P23SYS_IL_P24 SYS_LOB_SUBP32 LOB SUBPARTITION SYS_LOB_SUBP33 LOB SUBPARTITION SYS_LOB_SUBP35 LOB SUBPARTITION SYS_LOB_SUBP36 LOB SUBPARTITION SYS_LOB_P34 LOB PARTITION SYS_LOB_P31 LOB PARTITION LOB SYS_LOB C00003$$ LOB SYS_LOB_SUBP32 INDEX SUBPARTITION SYS_LOB_SUBP33 INDEX SUBPARTITION SYS_LOB_SUBP35 INDEX SUBPARTITION SYS_LOB_SUBP36 INDEX SUBPARTITION SYS_LOB_P44 INDEX PARTITION SYS_LOB_P41 INDEX PARTITION LOB SYS_IL C00003$$ LOB INDEX P1S1 TABLE SUBPARTITION P1S2 TABLE SUBPARTITION P2 TABLE PARTITION P1 TABLE PARTITION LOB T4 TABLE P2S1 TABLE SUBPARTITION P2S2 TABLE SUBPARTITION

© 2008 Julian Dyke juliandyke.com 19 Basic Files In Row versus Out-of-Line Storage  Values can be stored  in-line - in a row piece  out-of-line - in a separate LOB segment  ENABLE STORAGE IN ROW (default)  LOB value is stored within row if size is < 3964 bytes  LOB value is stored out-of-line if size > 3964 bytes  If in-line LOB grows to more than 3964 bytes  LOB value is moved out-of-line to LOB segment  If out-of-line LOB shrinks to less than 3964 bytes  LOB value remains out-of-line in LOB segment  DISABLE STORAGE IN ROW  LOB value is always stored out-of-line in LOB segment CREATE TABLE t1 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (ENABLE STORAGE IN ROW); CREATE TABLE t2 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (DISABLE STORAGE IN ROW);

© 2008 Julian Dyke juliandyke.com 20 Basic Files LOB Initialization  LOBs are initialized using built-in functions  EMPTY_BLOB ()  EMPTY_CLOB ()  Both functions create LOB locators  Minimum size is 36 bytes  Maximum size is 86 bytes CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, ); INSERT INTO t1 VALUES (c1,NULL); INSERT INTO t1 VALUES (c1,EMPTY_CLOB()); C2 occupies 0 bytes C2 occupies 37bytes

© 2008 Julian Dyke juliandyke.com 21 Basic Files LOB Initialization  LOB locator is initialized using EMPTY_CLOB() or EMPTY_BLOB()  For example: c d tab 0, row tl: 44 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 3] c col 1: [36] c d INSERT INTO t1 VALUES (c1,EMPTY_CLOB());  Block dump includes : LOBIDHeader Body Length LOB LengthVersionFlags

© 2008 Julian Dyke juliandyke.com 22 Basic Files LOB ID  LOB ID is a 10 byte number identifying individual instance of a LOB  Allocated when LOB value is created including EMPTY_CLOB() etc  Format is where  is a currently unknown 4-byte number (always 1)  is a 6-byte number generated from sequence SYS.IDGEN$  For example: INSERT INTO t1 VALUES (0,EMPTY_CLOB) SELECT sequence_owner, sequence_name, nextvalue, increment_by, cache_size FROM v$_sequences; Sequence OwnerSequence NameNext ValueIncrement ByCache Size SYSIDGEN1$ SELECT sequence_owner, sequence_name, nextvalue, increment_by, cache_size FROM v$_sequences; Sequence OwnerSequence NameNext ValueIncrement ByCache Size SYSIDGEN1$  LOB ID =(0x9219 = 37401)

© 2008 Julian Dyke juliandyke.com 23 Basic Files LOB Inline Storage  Example byte LOB - ENABLE STORAGE IN ROW c a7b 001c0900 0c ab 0, row tl: 54 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [48] c a 7b 00 1c c INSERT INTO t1 VALUES (c1, LPAD ('X',12,'X')); 0x0C = 12 0x1C = 28

© 2008 Julian Dyke juliandyke.com 24 Basic Files LOB Inline Storage  Example byte LOB - ENABLE STORAGE IN ROW c aad 01a tab 0, row tl: 444 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [436] c a ad 01 a INSERT INTO t1 VALUES (c1,LPAD ('X',400,'X')); 0x0190 = 400 0x01A0 = 416

© 2008 Julian Dyke juliandyke.com 25 Basic Files LOB Inline Storage  Example byte LOB - ENABLE STORAGE IN ROW c ca1 0f8c0900 0f7c tab 0, row tl: 4008 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [4000] c c a1 0f 8c f 7c INSERT INTO t1 VALUES (c1,LPAD ('X',3964,'X')); 0x0F7C = x0F8C = 3980

© 2008 Julian Dyke juliandyke.com 26 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c d f7d b5 tab 0, row tl: 46 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [40] c d f 7d b5 INSERT INTO t1 VALUES (c1,LPAD ('X',3965,'X')); 0x0014 = 200x0F7D bytes Address of first chunk 0x0F7D = 3965

© 2008 Julian Dyke juliandyke.com 27 Basic Files LOB Out-of-Line Storage  Out-of-line LOB data is stored in a PAGETABLE MANAGED LOB BLOCK  Each block has  a 56 byte header  a 4 byte trailer  For example for a 8192 byte block Trailer 4 bytes Header 56 bytes LOB Data 8132 bytes

© 2008 Julian Dyke juliandyke.com 28 CREATE DIRECTORY dir1 AS '/tmp'; CREATE TABLE t11 ( c1 NUMBER, c2 CLOB ); INSERT INTO t11 VALUES (0,EMPTY_CLOB()); Basic Files Out-of-Line Storage DECLARE l_bfile BFILE; l_clob CLOB; l_result NUMBER; BEGIN l_bfile := BFILENAME ('DIR1','dbms_stats.lst'); DBMS_LOB.OPEN (l_bfile); SELECT c2 INTO l_clob FROM t11 WHERE c1 = 0 FOR UPDATE; l_result := DBMS_LOB.GETLENGTH (l_bfile); DBMS_LOB.LOADFROMFILE (l_clob,l_bfile,l_result); DBMS_LOB.CLOSE (l_bfile); END;

© 2008 Julian Dyke juliandyke.com 29 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c dff f c d e e f a tl: 78 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [72] c d ff f c d e e f a INSERT INTO t1 VALUES (c1,BFILE...); 8 x 8132 = x1055 = chunks 0x34 = 52 DBA of LOB chunks 0x1055 bytes

© 2008 Julian Dyke juliandyke.com 30 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c c d e f c tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c c d e f c LOB Index exists but is empty 12 chunks 0 bytes 0x40 = 64 INSERT INTO t1 VALUES (c1,BFILE...); 12 x 8132 = 97584

© 2008 Julian Dyke juliandyke.com 31 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c c d e f c tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c c d e f c INSERT INTO t1 VALUES (c1,BFILE...); 12 chunks 1 byte 0x40 = 64 LOB Index is used for 13 th chunk

© 2008 Julian Dyke juliandyke.com 32 Basic Files LOB Out-of-Line Storage  Example 7 continued byte LOB - ENABLE STORAGE IN ROW  LOB Index f00 row#0[7982] flag: , lock: 2, len=50, data:(32): f col 0; len 10; (10): col 1; len 4; (4): c c Key Data First chunk number  LOB Index is always unique  Data is always 32 bytes LOBID

© 2008 Julian Dyke juliandyke.com 33 Basic Files LOB Out-of-Line Storage  Example byte LOB - ENABLE STORAGE IN ROW c ed e f a b c d e tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c ed e f a b c d e INSERT INTO t1 VALUES (c1,BFILE...); 0x17 = 23 chunks 12 chunks in row 11 chunks in index 0xe4 = 228 bytes

© 2008 Julian Dyke juliandyke.com 34 Basic Files LOB Out-of-Line Storage  Example 8 continued byte LOB - ENABLE STORAGE IN ROW  Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): f a b c d e col 0; len 10; (10): ed col 1; len 4; (4): c row#1[7932] flag: , lock: 2, len=50, data:(32): col 0; len 10; (10): ed col 1; len 4; (4): f a b c d e ed00 0c ed00 14 Key Data

© 2008 Julian Dyke juliandyke.com 35 Basic Files Chunk Size  Each internal LOB has a chunk size  Chunks can be manipulated individually  Default chunk size is block size  Maximum chunk size is for all tablespace block sizes  Chunk size is:  specified in bytes  rounded up to nearest block size CREATE TABLE t3 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CHUNK 32768);  Chunk size is reported in DBA_LOB.CHUNK  Chunk size is stored in SYS.LOB$.CHUNK  Although chunk size is specified and reported in bytes, it is stored in LOB$ in terms of tablespace blocks

© 2008 Julian Dyke juliandyke.com 36 Basic Files Chunk Size  Example byte LOB  CHUNK = 8192 (default) c dff f c d e e f a c d5 001c d tab 0, row tl: 54 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [48] c d5 00 1c d  CHUNK = (maximum) 4 x 8132 = x = x1055 =

© 2008 Julian Dyke juliandyke.com 37 Basic Files Chunk Size  In Oracle 11.1 maximum LOB size is determined by chunk size: CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, c3 CLOB, c4 CLOB ) LOB (c2) STORE AS (CHUNK 8192), LOB (c3) STORE AS (CHUNK 16384), LOB (c4) STORE AS (CHUNK 32768); SELECT dbms_lob.getchunksize (c2), dbms_lob.get_storage_limit (c2), dbms_lob.getchunksize (c3), dbms_lob.get_storage_limit (c3), dbms_lob.getchunksize (c4), dbms_lob.get_storage_limit (c4) FROM t1; Chunk SizeStorage Limit# Chunks ,926,674,042,9404,294,967, ,853,348,085,8804,294,967, ,706,696,171,7604,294,967,295

© 2008 Julian Dyke juliandyke.com 38 Basic Files Read Consistency  PCTVERSION clause  Specifies the amount of space that will be used for the LOB consistent- read mechanism  Affects reclamation of old copies of LOB data  Affects the ability to perform consistent read  Range is 0 to 100  Default value is 10 CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (PCTVERSION 20);  Reported in DBA_LOB.PCTVERSION  Can be set to 0 for read-only LOBs  PCTVERSION should be set to a low value if:  updates and reads are not concurrent  the LOB is written once and then read-only  PCTVERSION should be set to a high value if:  there are large numbers of queries  there is heavy read and write activity

© 2008 Julian Dyke juliandyke.com 39 Basic Files Read Consistency  Example byte LOB - update 4000 bytes starting at offset  PCTVERSION = 10 (default) DECLARE l_clob CLOB; l_str VARCHAR2(4000); BEGIN l_str := LPAD ('Y',4000,'Y'); SELECT c2 INTO l_clob FROM t19 WHERE c1 = 0; FOR UPDATE; DBMS_LOB.WRITE (l_clob,4000,10000,l_str); END; /

© 2008 Julian Dyke juliandyke.com 40 Basic Files Read Consistency  Example 9 (continued)  Before Update tl: 62 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [56] c f fd f ff fc c f fd f010003ff fc c f fd fe010003ff fc  After Update Updated Chunk

© 2008 Julian Dyke juliandyke.com 41 Basic Files Read Consistency  Example 9 (continued) row#0[7982] flag: , lock: 2, len=50, data:(32): fe col 0; len 10; (10): f1 b col 1; len 4; (4): fe fe00 LOB Index LOB Data fec f Key Data Free List 3FD3FE3FF4003FC 3FE 40F STOP

© 2008 Julian Dyke juliandyke.com 42 Basic Files Read Consistency  Example byte LOB - update 4000 bytes starting at offset  PCTVERSION = 10 DECLARE l_clob CLOB; l_str VARCHAR2(4000); BEGIN l_str := LPAD ('Y',4000,'Y'); SELECT c2 INTO l_clob FROM t19 WHERE c1 = 0; FOR UPDATE; DBMS_LOB.WRITE (l_clob,4000,30000,l_str); END; /

© 2008 Julian Dyke juliandyke.com 43 Basic Files Read Consistency  Example 10 (continued)  Before Update tl: 62 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [56] c bf fd f ff fe c bf fd f010003ff010003fe c f fd f010003ff fc  After Update Updated Chunk

© 2008 Julian Dyke juliandyke.com 44 Basic Files Read Consistency  Example 10 (continued) - Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): fc col 0; len 10; (10): fe c col 1; len 4; (4): fc fc fec f Key Data LOB Index LOB Data Free List 3FD3FE3FF400 3FC3FE40F4003FE3FC410 STOP

© 2008 Julian Dyke juliandyke.com 45 Basic Files Partial Deletion  Example byte LOB - Erase bytes from offset DECLARE l_clob CLOB; l_amount NUMBER := 18000; BEGIN SELECT c2 INTO l_clob FROM t22 WHERE c1 = 0 FOR UPDATE; DBMS_LOB.ERASE (l_clob,l_amount,12000); END;

© 2008 Julian Dyke juliandyke.com 46 Basic Files Partial Deletion  Example 11 continued byte LOB - Erase bytes from offset  Before ERASE c b d d e c f a b tab 0, row tl: 78 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [72] c b d d e c f a b c b d e f c f a b  After ERASE

© 2008 Julian Dyke juliandyke.com 47 Basic Files Partial Deletion  Example 11 continued byte LOB - Erase bytes from offset  After ERASE - Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): e f col 0; len 10; (10): a col 1; len 4; (4): e e f LOB Index LOB Data Free List Data 46D 46E47046C47F48047A46E46F470 47B46F47D47E STOP

© 2008 Julian Dyke juliandyke.com 48 Basic Files CACHE Clause  Specifies whether a LOB will be stored in the database buffer cache when values are read and written.  Allows LOB values to be shared amongst users.  Possible values are CACHE, CACHE READS and NOCACHE  Default value is NOCACHE CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE);  Reported in DBA_LOB.CACHE  Can be NO, CACHEREADS or YES CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS);

© 2008 Julian Dyke juliandyke.com 49 Basic Files CACHE versus NOCACHE  CACHE LOGGING Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time Wait Class enq: CF - contention 6, Other CPU time log file sync 69, Commit control file parallel write 31, System I/O log file parallel write 60, System I/O Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time Wait Class CPU time log file sync 79, Commit log file parallel write 66, System I/O SQL*Net more data from client 37, Network db file parallel write 5, System I/O  NOCACHE NOLOGGING

© 2008 Julian Dyke juliandyke.com 50 Basic Files LOGGING  Possible values are LOGGING (default) and NOLOGGING  If CACHE option is specified then LOGGING is mandatory  LOGGING/NOLOGGING cannot be specified without CACHE clause CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE NOLOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS NOLOGGING);  Reported in DBA_LOB.LOGGING  Can be NO or YES

© 2008 Julian Dyke juliandyke.com 51 Basic Files LOB Compression  LOBs can be compressed using the LZ_COMPRESS procedure in the UTL_COMPRESS package  A quality can be specified  1 - Fastest  9 - Slowest  The default quality is 6  Compression achieved is not very consistent File size in bytes Qualitydbms_stats.lstalert.log Uncompressed

© 2008 Julian Dyke juliandyke.com 52 Basic Files LOB Compression  Example DECLARE l_bfile BFILE; l_blob BLOB; l_clob CLOB; l_length NUMBER; l_quality NUMBER := 1; l_dest_offset NUMBER := 1; l_source_offset NUMBER := 1; l_blob_csid NUMBER := DBMS_LOB.DEFAULT_CSID; l_lang_context NUMBER := DBMS_LOB.DEFAULT_LANG_CTX; l_warning NUMBER; BEGIN l_bfile := BFILENAME ('DIR1','alert.log'); DBMS_LOB.OPEN (l_bfile); DBMS_LOB.CREATETEMPORARY (l_blob,FALSE); l_length := DBMS_LOB.GETLENGTH (l_bfile); UTL_COMPRESS.LZ_COMPRESS (l_bfile,l_blob,l_quality); SELECT c2 INTO l_clob FROM t34 WHERE c1 = 0 FOR UPDATE; DBMS_LOB.CONVERTTOCLOB (l_clob,l_blob,DBMS_LOB.LOBMAXSIZE, l_dest_offset,l_source_offset,l_blob_csid,l_lang_context,l_warning); END; CREATE DIRECTORY dir1 AS '/tmp'; CREATE TABLE t34 (c1 NUMBER, c2 CLOB); INSERT INTO t34 VALUES (0,EMPTY_CLOB ());

© 2008 Julian Dyke juliandyke.com 53 Secure Files

© 2008 Julian Dyke juliandyke.com 54 Secure Files Overview  In Oracle 11.1 and above there are two types of LOB  STORE AS BASICFILE (default)  STORE AS SECUREFILE  STORE AS BASICFILE  Default  Behaves as Oracle 10.2 and below  STORE AS SECUREFILE  Oracle 11.1 and above  Supports several new features including  LOB compression  LOB encryption  LOB deduplication  Metadata only logging  Configurable retention time

© 2008 Julian Dyke juliandyke.com 55 SecureFiles Storage  Can only be created in an ASSM tablespace  New block types supporting SecureFiles include: DescriptionBlock Type NGLOB: Extent Map0x3c NGLOB: Hash Bucket0x3d NGLOB: Committed Free Space0x3e NGLOB: Segment Header0x3f NGLOB: Lob Extent Header0x40 NGLOB: Persistent Undo0x41  LOB Data Segments are still stored in blocks of type trans_data (block type 6)  Note  CHUNK is advisory - only maintained for backward compatibility  PCTVERSION not supported - use RETENTION instead

© 2008 Julian Dyke juliandyke.com 56 Secure File Initialization  EMPTY_CLOB ()  Basic File c a tl: 36 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [30] c a c d  Secure File

© 2008 Julian Dyke juliandyke.com 57 Secure File LOB Locator  Example DBMS_STATS.LST bytes  Basic File c b 001f e d tl: 57 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [51] c b 00 1f e d c dff f c d e e f a  Secure File DBA of first block in extent # Blocks in extent 0x10E75 = 69237

© 2008 Julian Dyke juliandyke.com 58 Secure Files Lob Blocks  Out-of-line LOB data is stored in a trans data block (block type 6)  Each block has  a 128 byte header  a 4 byte trailer  For example for a 8192 byte block Trailer 4 bytes Header 128 bytes LOB Data 8060 bytes

© 2008 Julian Dyke juliandyke.com 59 Secure Files LOB Compression  Oracle 11.1 and above  Enables server-side LOB compression  Random read/write access is still possible  Independent of table compression / index compression  Only valid for SecureFile LOBs  Valid values are  COMPRESS  COMPRESS MEDIUM  COMPRESS HIGH  NOCOMPRESS  COMPRESS MEDIUM  Faster (less resource)  COMPRESS HIGH  Better compression (less storage)  Individual LOBs can be compressed using DBMS_LOB.SETOPTIONS ()

© 2008 Julian Dyke juliandyke.com 60 Secure Files LOB Compression  Compressed size of LOB is not reported in Oracle 11.1  To calculate compressed size count number of extents  In block dump  Using DBMS_LOBUTIL.GETINODE  Size will be an (over) estimate  Example - alert.log - sizes in bytes Uncompressed Size COMPRESS MEDIUM72540 COMPRESS HIGH40300  SecureFile LOB Compression much better than UTLCOMPRESS package  For example best compression achieved for same file using LZ_COMPRESS was bytes

© 2008 Julian Dyke juliandyke.com 61 Secure Files DBMS_LOBUTIL  Unsupported package that describes SecureFile LOBs  Defined in $ORACLE_HOME/rdbms/admin/dbmslobu.sql  Subroutines include  GETINODE  GETLOBMAP  GETEXTENTS  GETINODE  Reports LOB header  GETLOBMAP  Reports LOB extent header  Numeric parameter is extent number (0..N-1)  GETEXTENTS  Reports LOB extents in more detail  Pipelined function - takes REF CURSOR as parameter

© 2008 Julian Dyke juliandyke.com 62 Secure Files DBMS_LOBUTIL DECLARE l_clob CLOB; l_inode DBMS_LOBUTIL_INODE_T; l_lobmap DBMS_LOBUTIL_LOBMAP_T; BEGIN SELECT c2 INTO l_clob FROM t29 WHERE c1 = 0 FOR UPDATE; l_inode := DBMS_LOBUTIL.GETINODE (l_clob); DBMS_OUTPUT.PUT_LINE ('LOBID '||RAWTOHEX (l_inode.lobid)); DBMS_OUTPUT.PUT_LINE (' flags '||l_inode.flags); DBMS_OUTPUT.PUT_LINE (' length '||l_inode.length); DBMS_OUTPUT.PUT_LINE (' version '||l_inode.version); DBMS_OUTPUT.PUT_LINE (' extents '||l_inode.extents); FOR i IN 0..l_inode.extents - 1 LOOP l_res := DBMS_LOBUTIL.GETLOBMAP (l_clob,i); DBMS_OUTPUT.PUT_LINE ('Extent# '||i); DBMS_OUTPUT.PUT_LINE (' rdba '||l_res.rdba); DBMS_OUTPUT.PUT_LINE (' nblks '||l_res.nblks); DBMS_OUTPUT.PUT_LINE (' offset '||l_res.offset); DBMS_OUTPUT.PUT_LINE (' length '||l_res.length); END LOOP; END;

© 2008 Julian Dyke juliandyke.com 63 References  Oracle Database Application Developer's Guide - Large Objects 10g Release 2 (10.2) - Part Number B  Oracle Database SecureFiles and Large Objects Developers Guide 11g Release 1 (11.1) - Part Number B

© 2008 Julian Dyke juliandyke.com 64 Any Questions ?