1 © 2008 Julian Dyke LOB Internals Julian Dyke Independent Consultant juliandyke.com Web Version - December 2008
© 2008 Julian Dyke juliandyke.com 2 Objectives 1. Understand how LOBs use storage in the Oracle database 2. Consider options to optimize LOB performance in terms of: Reads Writes Impact on physical I/O Impact on buffer cache
© 2008 Julian Dyke juliandyke.com 3 Agenda Basic Files Secure Files
© 2008 Julian Dyke juliandyke.com 4 Basic Files
© 2008 Julian Dyke juliandyke.com 5 Basic Files Overview Introduced in Oracle 8.0 Known as Basic Files in Oracle 11.1 and above Intended to replace LONG columns Can be used to store large blocks of unstructured data e.g.: text graphics video clips sound waveforms XML documents Can store either character or binary data Can be stored within database (internal) outside database (external)
© 2008 Julian Dyke juliandyke.com 6 Basic Files LOB Types There are four types of LOB: BLOB - Binary Large Object Stored within database Contains raw data CLOB - Character Large Object Stored within database Supports database character set NCLOB - NLS Character Large Object Stored within database Supports NLS character set BFILE - Binary File Large Object Stored in a binary file outside the database Pointed to by a file locator within the database
© 2008 Julian Dyke juliandyke.com 7 Basic Files Internal LOBs Stored outside of database BLOB, CLOB or NCLOB Maximum size is dependent on version and chunk size Support features such as: concurrency redo logging and recovery transactions with commits and rollbacks For each row with an internal LOB column, the LOB can: Be NULL Point to an empty BLOB, CLOB or NCLOB Point to a BLOB, CLOB or NCLOB value
© 2008 Julian Dyke juliandyke.com 8 Basic Files LOB columns versus LONG columns LOBs are intended to replace LONG columns: Oracle recommends converting LONG columns to LOB columns LONG columns still used by many applications LONG columns still used in Oracle 11.1 data dictionary tables including: COL$, TABPART$, INDPART$, TRIGGER$, VIEW$ and CDEF$ Therefore it is unlikely LONG columns will be desupported in the near future LONGs Single column per table Maximum size 2GB Data always stored in-line Cannot be an object attribute Cannot be partitioned Cannot be used in IOTs Cannot be replicated Access is sequential Partial PL/SQL support LOBs Multiple columns per table Maximum size 4GB or more Data stored in-line or out-of-line Can be an object attribute Can be partitioned Can be used in IOTs Can be replicated Access can be is random Full PL/SQL support
© 2008 Julian Dyke juliandyke.com 9 Basic Files DBMS_LOB Package BLOB and CLOB Subroutines GETOPTIONS (11.1+) GET_STORAGE_LIMIT INSTR ISSECUREFILE ISTEMPORARY ISOPEN LOADFROMFILE LOADBLOBFROMFILE LOADCLOBFROMFILE OPEN READ SETOPTIONS (11.1+) SUBSTR TRIM WRITE WRITEAPPEND APPEND CLOSE COMPARE CONVERTTOBLOB CONVERTTOCLOB COPY CREATETEMPORARY ERASE FRAGMENT_DELETE (11.1+) FRAGMENT_INSERT (11.1+) FRAGMENT_MOVE (11.1+) FRAGMENT_REPLACE (11.1+) FREETEMPORARY GETCHUNKSIZE GET_DEDUPLICATE_REGIONS (11.1+) GETLENGTH
© 2008 Julian Dyke juliandyke.com 10 Basic Files DBMS_LOB Package BFILE Subroutines CLOSE COMPARE FILECLOSE FILECLOSEALL FILEEXISTS FILEGETNAME FILEISOPEN FILEOPEN GETLENGTH INSTR ISOPEN OPEN READ SUBSTR
© 2008 Julian Dyke juliandyke.com 11 Basic Files DBA_LOBS View Column NameData TypeComments OWNERVARCHAR2(30) TABLE_NAMEVARCHAR2(30) COLUMN_NAMEVARCHAR2(4000) SEGMENT_NAMEVARCHAR2(30) TABLESPACE_NAMEVARCHAR2(30) INDEX_NAMEVARCHAR2(30) CHUNKNUMBER PCTVERSIONNUMBER RETENTIONNUMBER FREEPOOLSNUMBER CACHEVARCHAR2(10) LOGGINGVARCHAR2(7) ENCRYPTVARCHAR2(4)Oracle 11.1 and above COMPRESSIONVARCHAR2(6)Oracle 11.1 and above DEDUPLICATIONVARCHAR2(15)Oracle 11.1 and above IN_ROWVARCHAR2(3) FORMATVARCHAR2(15) PARTITIONEDVARCHAR2(3) SECUREFILEVARCHAR2(3)Oracle 11.1 and above
© 2008 Julian Dyke juliandyke.com 12 Basic Files LOB Objects If a table includes an internal LOB column, the following objects will be created LOB index LOB data Each object has a separate segment For example: CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, ); LOB SYS_IL C00002$$ LOB INDEX LOB SYS_LOB C00002$$ LOB DATA LOB T1 TABLE
© 2008 Julian Dyke juliandyke.com 13 Basic Files LOB Segments Every LOB has a LOB data segment and a LOB index segment LOB data segment is reported by DBA_OBJECTS Segment name is system generated e.g. SYS_LOB C00002$$ where: is the OBJECT_ID of the table is the column number of the LOB column within the table LOB index is not reported by DBA_OBJECTS Index is still defined in SYS.OBJ$ and SYS.IND$ Index name is reported by DBA_LOBS e.g. SYS_IL C00002$$ where: is the OBJECT_ID of the table is the column number of the LOB column within the table
© 2008 Julian Dyke juliandyke.com 14 Basic Files LOB Objects One LOB index segment and one LOB data segment will be created for each LOB column For example: CREATE TABLE t2 ( c1 NUMBER, c2 CLOB, c3 CLOB ); LOB SYS_IL C00002$$ LOB INDEX LOB SYS_IL C00003$$ LOB INDEX LOB SYS_LOB C00003$$ LOB DATA LOB SYS_LOB C00002$$ LOB DATA LOB T2 TABLE
© 2008 Julian Dyke juliandyke.com 15 Basic Files Partitioned LOBs Oracle supports partitioned and sub-partitioned LOBS in Oracle and above LOBs are supported in partitioned IOTs in Oracle and above LOB and and LOB index segments are equi-partitioned with base table Partitioning key cannot contain a LOB column CREATE TABLE t3 ( c1 NUMBER, c2 CLOB ) PARTITION BY RANGE (c1) ( PARTITION p1 VALUES LESS THAN (100), PARTITION p2 VALUES LESS THAN (200) );
© 2008 Julian Dyke juliandyke.com 16 Basic Files Partitioned LOBs SYS_LOB_P22 LOB PARTITION SYS_LOB_P21 LOB PARTITION LOB SYS_LOB C00002$$ LOB SYS_LOB_P24 INDEX PARTITION SYS_LOB_P23 INDEX PARTITION LOB SYS_IL C00002$$ LOB INDEX P2 TABLE PARTITION P1 TABLE PARTITION LOB T3 TABLE
© 2008 Julian Dyke juliandyke.com 17 Basic Files Sub Partitioned LOBs CREATE TABLE t4 ( c1 NUMBER, c2 NUMBER, c3 CLOB ) PARTITION BY RANGE (c1) ( PARTITION p1 VALUES LESS THAN (100) ( SUBPARTITION p1s1 VALUES LESS THAN (10), SUBPARTITION p1s2 VALUES LESS THAN (20) ), PARTITION p2 VALUES LESS THAN (200) ( SUBPARTITION p2s1 VALUES LESS THAN (10), SUBPARTITION p2s2 VALUES LESS THAN (20) ) );
© 2008 Julian Dyke juliandyke.com 18 Basic Files Sub Partitioned LOBs SYS_IL_P23SYS_IL_P24 SYS_LOB_SUBP32 LOB SUBPARTITION SYS_LOB_SUBP33 LOB SUBPARTITION SYS_LOB_SUBP35 LOB SUBPARTITION SYS_LOB_SUBP36 LOB SUBPARTITION SYS_LOB_P34 LOB PARTITION SYS_LOB_P31 LOB PARTITION LOB SYS_LOB C00003$$ LOB SYS_LOB_SUBP32 INDEX SUBPARTITION SYS_LOB_SUBP33 INDEX SUBPARTITION SYS_LOB_SUBP35 INDEX SUBPARTITION SYS_LOB_SUBP36 INDEX SUBPARTITION SYS_LOB_P44 INDEX PARTITION SYS_LOB_P41 INDEX PARTITION LOB SYS_IL C00003$$ LOB INDEX P1S1 TABLE SUBPARTITION P1S2 TABLE SUBPARTITION P2 TABLE PARTITION P1 TABLE PARTITION LOB T4 TABLE P2S1 TABLE SUBPARTITION P2S2 TABLE SUBPARTITION
© 2008 Julian Dyke juliandyke.com 19 Basic Files In Row versus Out-of-Line Storage Values can be stored in-line - in a row piece out-of-line - in a separate LOB segment ENABLE STORAGE IN ROW (default) LOB value is stored within row if size is < 3964 bytes LOB value is stored out-of-line if size > 3964 bytes If in-line LOB grows to more than 3964 bytes LOB value is moved out-of-line to LOB segment If out-of-line LOB shrinks to less than 3964 bytes LOB value remains out-of-line in LOB segment DISABLE STORAGE IN ROW LOB value is always stored out-of-line in LOB segment CREATE TABLE t1 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (ENABLE STORAGE IN ROW); CREATE TABLE t2 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (DISABLE STORAGE IN ROW);
© 2008 Julian Dyke juliandyke.com 20 Basic Files LOB Initialization LOBs are initialized using built-in functions EMPTY_BLOB () EMPTY_CLOB () Both functions create LOB locators Minimum size is 36 bytes Maximum size is 86 bytes CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, ); INSERT INTO t1 VALUES (c1,NULL); INSERT INTO t1 VALUES (c1,EMPTY_CLOB()); C2 occupies 0 bytes C2 occupies 37bytes
© 2008 Julian Dyke juliandyke.com 21 Basic Files LOB Initialization LOB locator is initialized using EMPTY_CLOB() or EMPTY_BLOB() For example: c d tab 0, row tl: 44 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 3] c col 1: [36] c d INSERT INTO t1 VALUES (c1,EMPTY_CLOB()); Block dump includes : LOBIDHeader Body Length LOB LengthVersionFlags
© 2008 Julian Dyke juliandyke.com 22 Basic Files LOB ID LOB ID is a 10 byte number identifying individual instance of a LOB Allocated when LOB value is created including EMPTY_CLOB() etc Format is where is a currently unknown 4-byte number (always 1) is a 6-byte number generated from sequence SYS.IDGEN$ For example: INSERT INTO t1 VALUES (0,EMPTY_CLOB) SELECT sequence_owner, sequence_name, nextvalue, increment_by, cache_size FROM v$_sequences; Sequence OwnerSequence NameNext ValueIncrement ByCache Size SYSIDGEN1$ SELECT sequence_owner, sequence_name, nextvalue, increment_by, cache_size FROM v$_sequences; Sequence OwnerSequence NameNext ValueIncrement ByCache Size SYSIDGEN1$ LOB ID =(0x9219 = 37401)
© 2008 Julian Dyke juliandyke.com 23 Basic Files LOB Inline Storage Example byte LOB - ENABLE STORAGE IN ROW c a7b 001c0900 0c ab 0, row tl: 54 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [48] c a 7b 00 1c c INSERT INTO t1 VALUES (c1, LPAD ('X',12,'X')); 0x0C = 12 0x1C = 28
© 2008 Julian Dyke juliandyke.com 24 Basic Files LOB Inline Storage Example byte LOB - ENABLE STORAGE IN ROW c aad 01a tab 0, row tl: 444 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [436] c a ad 01 a INSERT INTO t1 VALUES (c1,LPAD ('X',400,'X')); 0x0190 = 400 0x01A0 = 416
© 2008 Julian Dyke juliandyke.com 25 Basic Files LOB Inline Storage Example byte LOB - ENABLE STORAGE IN ROW c ca1 0f8c0900 0f7c tab 0, row tl: 4008 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [4000] c c a1 0f 8c f 7c INSERT INTO t1 VALUES (c1,LPAD ('X',3964,'X')); 0x0F7C = x0F8C = 3980
© 2008 Julian Dyke juliandyke.com 26 Basic Files LOB Out-of-Line Storage Example byte LOB - ENABLE STORAGE IN ROW c d f7d b5 tab 0, row tl: 46 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [40] c d f 7d b5 INSERT INTO t1 VALUES (c1,LPAD ('X',3965,'X')); 0x0014 = 200x0F7D bytes Address of first chunk 0x0F7D = 3965
© 2008 Julian Dyke juliandyke.com 27 Basic Files LOB Out-of-Line Storage Out-of-line LOB data is stored in a PAGETABLE MANAGED LOB BLOCK Each block has a 56 byte header a 4 byte trailer For example for a 8192 byte block Trailer 4 bytes Header 56 bytes LOB Data 8132 bytes
© 2008 Julian Dyke juliandyke.com 28 CREATE DIRECTORY dir1 AS '/tmp'; CREATE TABLE t11 ( c1 NUMBER, c2 CLOB ); INSERT INTO t11 VALUES (0,EMPTY_CLOB()); Basic Files Out-of-Line Storage DECLARE l_bfile BFILE; l_clob CLOB; l_result NUMBER; BEGIN l_bfile := BFILENAME ('DIR1','dbms_stats.lst'); DBMS_LOB.OPEN (l_bfile); SELECT c2 INTO l_clob FROM t11 WHERE c1 = 0 FOR UPDATE; l_result := DBMS_LOB.GETLENGTH (l_bfile); DBMS_LOB.LOADFROMFILE (l_clob,l_bfile,l_result); DBMS_LOB.CLOSE (l_bfile); END;
© 2008 Julian Dyke juliandyke.com 29 Basic Files LOB Out-of-Line Storage Example byte LOB - ENABLE STORAGE IN ROW c dff f c d e e f a tl: 78 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [72] c d ff f c d e e f a INSERT INTO t1 VALUES (c1,BFILE...); 8 x 8132 = x1055 = chunks 0x34 = 52 DBA of LOB chunks 0x1055 bytes
© 2008 Julian Dyke juliandyke.com 30 Basic Files LOB Out-of-Line Storage Example byte LOB - ENABLE STORAGE IN ROW c c d e f c tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c c d e f c LOB Index exists but is empty 12 chunks 0 bytes 0x40 = 64 INSERT INTO t1 VALUES (c1,BFILE...); 12 x 8132 = 97584
© 2008 Julian Dyke juliandyke.com 31 Basic Files LOB Out-of-Line Storage Example byte LOB - ENABLE STORAGE IN ROW c c d e f c tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c c d e f c INSERT INTO t1 VALUES (c1,BFILE...); 12 chunks 1 byte 0x40 = 64 LOB Index is used for 13 th chunk
© 2008 Julian Dyke juliandyke.com 32 Basic Files LOB Out-of-Line Storage Example 7 continued byte LOB - ENABLE STORAGE IN ROW LOB Index f00 row#0[7982] flag: , lock: 2, len=50, data:(32): f col 0; len 10; (10): col 1; len 4; (4): c c Key Data First chunk number LOB Index is always unique Data is always 32 bytes LOBID
© 2008 Julian Dyke juliandyke.com 33 Basic Files LOB Out-of-Line Storage Example byte LOB - ENABLE STORAGE IN ROW c ed e f a b c d e tab 0, row tl: 90 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [84] c ed e f a b c d e INSERT INTO t1 VALUES (c1,BFILE...); 0x17 = 23 chunks 12 chunks in row 11 chunks in index 0xe4 = 228 bytes
© 2008 Julian Dyke juliandyke.com 34 Basic Files LOB Out-of-Line Storage Example 8 continued byte LOB - ENABLE STORAGE IN ROW Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): f a b c d e col 0; len 10; (10): ed col 1; len 4; (4): c row#1[7932] flag: , lock: 2, len=50, data:(32): col 0; len 10; (10): ed col 1; len 4; (4): f a b c d e ed00 0c ed00 14 Key Data
© 2008 Julian Dyke juliandyke.com 35 Basic Files Chunk Size Each internal LOB has a chunk size Chunks can be manipulated individually Default chunk size is block size Maximum chunk size is for all tablespace block sizes Chunk size is: specified in bytes rounded up to nearest block size CREATE TABLE t3 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CHUNK 32768); Chunk size is reported in DBA_LOB.CHUNK Chunk size is stored in SYS.LOB$.CHUNK Although chunk size is specified and reported in bytes, it is stored in LOB$ in terms of tablespace blocks
© 2008 Julian Dyke juliandyke.com 36 Basic Files Chunk Size Example byte LOB CHUNK = 8192 (default) c dff f c d e e f a c d5 001c d tab 0, row tl: 54 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [48] c d5 00 1c d CHUNK = (maximum) 4 x 8132 = x = x1055 =
© 2008 Julian Dyke juliandyke.com 37 Basic Files Chunk Size In Oracle 11.1 maximum LOB size is determined by chunk size: CREATE TABLE t1 ( c1 NUMBER, c2 CLOB, c3 CLOB, c4 CLOB ) LOB (c2) STORE AS (CHUNK 8192), LOB (c3) STORE AS (CHUNK 16384), LOB (c4) STORE AS (CHUNK 32768); SELECT dbms_lob.getchunksize (c2), dbms_lob.get_storage_limit (c2), dbms_lob.getchunksize (c3), dbms_lob.get_storage_limit (c3), dbms_lob.getchunksize (c4), dbms_lob.get_storage_limit (c4) FROM t1; Chunk SizeStorage Limit# Chunks ,926,674,042,9404,294,967, ,853,348,085,8804,294,967, ,706,696,171,7604,294,967,295
© 2008 Julian Dyke juliandyke.com 38 Basic Files Read Consistency PCTVERSION clause Specifies the amount of space that will be used for the LOB consistent- read mechanism Affects reclamation of old copies of LOB data Affects the ability to perform consistent read Range is 0 to 100 Default value is 10 CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (PCTVERSION 20); Reported in DBA_LOB.PCTVERSION Can be set to 0 for read-only LOBs PCTVERSION should be set to a low value if: updates and reads are not concurrent the LOB is written once and then read-only PCTVERSION should be set to a high value if: there are large numbers of queries there is heavy read and write activity
© 2008 Julian Dyke juliandyke.com 39 Basic Files Read Consistency Example byte LOB - update 4000 bytes starting at offset PCTVERSION = 10 (default) DECLARE l_clob CLOB; l_str VARCHAR2(4000); BEGIN l_str := LPAD ('Y',4000,'Y'); SELECT c2 INTO l_clob FROM t19 WHERE c1 = 0; FOR UPDATE; DBMS_LOB.WRITE (l_clob,4000,10000,l_str); END; /
© 2008 Julian Dyke juliandyke.com 40 Basic Files Read Consistency Example 9 (continued) Before Update tl: 62 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [56] c f fd f ff fc c f fd f010003ff fc c f fd fe010003ff fc After Update Updated Chunk
© 2008 Julian Dyke juliandyke.com 41 Basic Files Read Consistency Example 9 (continued) row#0[7982] flag: , lock: 2, len=50, data:(32): fe col 0; len 10; (10): f1 b col 1; len 4; (4): fe fe00 LOB Index LOB Data fec f Key Data Free List 3FD3FE3FF4003FC 3FE 40F STOP
© 2008 Julian Dyke juliandyke.com 42 Basic Files Read Consistency Example byte LOB - update 4000 bytes starting at offset PCTVERSION = 10 DECLARE l_clob CLOB; l_str VARCHAR2(4000); BEGIN l_str := LPAD ('Y',4000,'Y'); SELECT c2 INTO l_clob FROM t19 WHERE c1 = 0; FOR UPDATE; DBMS_LOB.WRITE (l_clob,4000,30000,l_str); END; /
© 2008 Julian Dyke juliandyke.com 43 Basic Files Read Consistency Example 10 (continued) Before Update tl: 62 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [56] c bf fd f ff fe c bf fd f010003ff010003fe c f fd f010003ff fc After Update Updated Chunk
© 2008 Julian Dyke juliandyke.com 44 Basic Files Read Consistency Example 10 (continued) - Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): fc col 0; len 10; (10): fe c col 1; len 4; (4): fc fc fec f Key Data LOB Index LOB Data Free List 3FD3FE3FF400 3FC3FE40F4003FE3FC410 STOP
© 2008 Julian Dyke juliandyke.com 45 Basic Files Partial Deletion Example byte LOB - Erase bytes from offset DECLARE l_clob CLOB; l_amount NUMBER := 18000; BEGIN SELECT c2 INTO l_clob FROM t22 WHERE c1 = 0 FOR UPDATE; DBMS_LOB.ERASE (l_clob,l_amount,12000); END;
© 2008 Julian Dyke juliandyke.com 46 Basic Files Partial Deletion Example 11 continued byte LOB - Erase bytes from offset Before ERASE c b d d e c f a b tab 0, row tl: 78 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [72] c b d d e c f a b c b d e f c f a b After ERASE
© 2008 Julian Dyke juliandyke.com 47 Basic Files Partial Deletion Example 11 continued byte LOB - Erase bytes from offset After ERASE - Lob Index row#0[7982] flag: , lock: 2, len=50, data:(32): e f col 0; len 10; (10): a col 1; len 4; (4): e e f LOB Index LOB Data Free List Data 46D 46E47046C47F48047A46E46F470 47B46F47D47E STOP
© 2008 Julian Dyke juliandyke.com 48 Basic Files CACHE Clause Specifies whether a LOB will be stored in the database buffer cache when values are read and written. Allows LOB values to be shared amongst users. Possible values are CACHE, CACHE READS and NOCACHE Default value is NOCACHE CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE); Reported in DBA_LOB.CACHE Can be NO, CACHEREADS or YES CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS);
© 2008 Julian Dyke juliandyke.com 49 Basic Files CACHE versus NOCACHE CACHE LOGGING Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time Wait Class enq: CF - contention 6, Other CPU time log file sync 69, Commit control file parallel write 31, System I/O log file parallel write 60, System I/O Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time Wait Class CPU time log file sync 79, Commit log file parallel write 66, System I/O SQL*Net more data from client 37, Network db file parallel write 5, System I/O NOCACHE NOLOGGING
© 2008 Julian Dyke juliandyke.com 50 Basic Files LOGGING Possible values are LOGGING (default) and NOLOGGING If CACHE option is specified then LOGGING is mandatory LOGGING/NOLOGGING cannot be specified without CACHE clause CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE NOLOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (NOCACHE LOGGING); CREATE TABLE t4 (c1 NUMBER, c2 CLOB) LOB (c2) STORE AS (CACHE READS NOLOGGING); Reported in DBA_LOB.LOGGING Can be NO or YES
© 2008 Julian Dyke juliandyke.com 51 Basic Files LOB Compression LOBs can be compressed using the LZ_COMPRESS procedure in the UTL_COMPRESS package A quality can be specified 1 - Fastest 9 - Slowest The default quality is 6 Compression achieved is not very consistent File size in bytes Qualitydbms_stats.lstalert.log Uncompressed
© 2008 Julian Dyke juliandyke.com 52 Basic Files LOB Compression Example DECLARE l_bfile BFILE; l_blob BLOB; l_clob CLOB; l_length NUMBER; l_quality NUMBER := 1; l_dest_offset NUMBER := 1; l_source_offset NUMBER := 1; l_blob_csid NUMBER := DBMS_LOB.DEFAULT_CSID; l_lang_context NUMBER := DBMS_LOB.DEFAULT_LANG_CTX; l_warning NUMBER; BEGIN l_bfile := BFILENAME ('DIR1','alert.log'); DBMS_LOB.OPEN (l_bfile); DBMS_LOB.CREATETEMPORARY (l_blob,FALSE); l_length := DBMS_LOB.GETLENGTH (l_bfile); UTL_COMPRESS.LZ_COMPRESS (l_bfile,l_blob,l_quality); SELECT c2 INTO l_clob FROM t34 WHERE c1 = 0 FOR UPDATE; DBMS_LOB.CONVERTTOCLOB (l_clob,l_blob,DBMS_LOB.LOBMAXSIZE, l_dest_offset,l_source_offset,l_blob_csid,l_lang_context,l_warning); END; CREATE DIRECTORY dir1 AS '/tmp'; CREATE TABLE t34 (c1 NUMBER, c2 CLOB); INSERT INTO t34 VALUES (0,EMPTY_CLOB ());
© 2008 Julian Dyke juliandyke.com 53 Secure Files
© 2008 Julian Dyke juliandyke.com 54 Secure Files Overview In Oracle 11.1 and above there are two types of LOB STORE AS BASICFILE (default) STORE AS SECUREFILE STORE AS BASICFILE Default Behaves as Oracle 10.2 and below STORE AS SECUREFILE Oracle 11.1 and above Supports several new features including LOB compression LOB encryption LOB deduplication Metadata only logging Configurable retention time
© 2008 Julian Dyke juliandyke.com 55 SecureFiles Storage Can only be created in an ASSM tablespace New block types supporting SecureFiles include: DescriptionBlock Type NGLOB: Extent Map0x3c NGLOB: Hash Bucket0x3d NGLOB: Committed Free Space0x3e NGLOB: Segment Header0x3f NGLOB: Lob Extent Header0x40 NGLOB: Persistent Undo0x41 LOB Data Segments are still stored in blocks of type trans_data (block type 6) Note CHUNK is advisory - only maintained for backward compatibility PCTVERSION not supported - use RETENTION instead
© 2008 Julian Dyke juliandyke.com 56 Secure File Initialization EMPTY_CLOB () Basic File c a tl: 36 fb: --H-FL-- lb: 0x1 cc: 2 col 0: [ 1] 80 col 1: [30] c a c d Secure File
© 2008 Julian Dyke juliandyke.com 57 Secure File LOB Locator Example DBMS_STATS.LST bytes Basic File c b 001f e d tl: 57 fb: --H-FL-- lb: 0x2 cc: 2 col 0: [ 1] 80 col 1: [51] c b 00 1f e d c dff f c d e e f a Secure File DBA of first block in extent # Blocks in extent 0x10E75 = 69237
© 2008 Julian Dyke juliandyke.com 58 Secure Files Lob Blocks Out-of-line LOB data is stored in a trans data block (block type 6) Each block has a 128 byte header a 4 byte trailer For example for a 8192 byte block Trailer 4 bytes Header 128 bytes LOB Data 8060 bytes
© 2008 Julian Dyke juliandyke.com 59 Secure Files LOB Compression Oracle 11.1 and above Enables server-side LOB compression Random read/write access is still possible Independent of table compression / index compression Only valid for SecureFile LOBs Valid values are COMPRESS COMPRESS MEDIUM COMPRESS HIGH NOCOMPRESS COMPRESS MEDIUM Faster (less resource) COMPRESS HIGH Better compression (less storage) Individual LOBs can be compressed using DBMS_LOB.SETOPTIONS ()
© 2008 Julian Dyke juliandyke.com 60 Secure Files LOB Compression Compressed size of LOB is not reported in Oracle 11.1 To calculate compressed size count number of extents In block dump Using DBMS_LOBUTIL.GETINODE Size will be an (over) estimate Example - alert.log - sizes in bytes Uncompressed Size COMPRESS MEDIUM72540 COMPRESS HIGH40300 SecureFile LOB Compression much better than UTLCOMPRESS package For example best compression achieved for same file using LZ_COMPRESS was bytes
© 2008 Julian Dyke juliandyke.com 61 Secure Files DBMS_LOBUTIL Unsupported package that describes SecureFile LOBs Defined in $ORACLE_HOME/rdbms/admin/dbmslobu.sql Subroutines include GETINODE GETLOBMAP GETEXTENTS GETINODE Reports LOB header GETLOBMAP Reports LOB extent header Numeric parameter is extent number (0..N-1) GETEXTENTS Reports LOB extents in more detail Pipelined function - takes REF CURSOR as parameter
© 2008 Julian Dyke juliandyke.com 62 Secure Files DBMS_LOBUTIL DECLARE l_clob CLOB; l_inode DBMS_LOBUTIL_INODE_T; l_lobmap DBMS_LOBUTIL_LOBMAP_T; BEGIN SELECT c2 INTO l_clob FROM t29 WHERE c1 = 0 FOR UPDATE; l_inode := DBMS_LOBUTIL.GETINODE (l_clob); DBMS_OUTPUT.PUT_LINE ('LOBID '||RAWTOHEX (l_inode.lobid)); DBMS_OUTPUT.PUT_LINE (' flags '||l_inode.flags); DBMS_OUTPUT.PUT_LINE (' length '||l_inode.length); DBMS_OUTPUT.PUT_LINE (' version '||l_inode.version); DBMS_OUTPUT.PUT_LINE (' extents '||l_inode.extents); FOR i IN 0..l_inode.extents - 1 LOOP l_res := DBMS_LOBUTIL.GETLOBMAP (l_clob,i); DBMS_OUTPUT.PUT_LINE ('Extent# '||i); DBMS_OUTPUT.PUT_LINE (' rdba '||l_res.rdba); DBMS_OUTPUT.PUT_LINE (' nblks '||l_res.nblks); DBMS_OUTPUT.PUT_LINE (' offset '||l_res.offset); DBMS_OUTPUT.PUT_LINE (' length '||l_res.length); END LOOP; END;
© 2008 Julian Dyke juliandyke.com 63 References Oracle Database Application Developer's Guide - Large Objects 10g Release 2 (10.2) - Part Number B Oracle Database SecureFiles and Large Objects Developers Guide 11g Release 1 (11.1) - Part Number B
© 2008 Julian Dyke juliandyke.com 64 Any Questions ?