Working on the Chain Gang Utilizing Oracle for Offchain data storage Eric Herzog CMO Vice President, Worldwide Storage Channels IBM Storage Systems and Software-Defined Infrastructure herzog@us.ibm.com @zoginstor

Introduction Blockchain’s relationship to offchain storage is similar to fact and dimension tables in a star schema The Blockchain’s is the Fact table and the offchain storage the dimension tables

Introduction In a Data Warehouse the Fact table stores a row of foreign keys pointing to the dimension tables and the “facts” that result from that intersection of data sources. The blockchain transaction should only store the needed hash IDs, transaction codes and intersection data such as sums, amounts and aggregations resulting from the transaction Offchain data becomes the dimension tables, storing all detail data

Just in case… A chronologically ordered, cryptographically signature linked series of blocks containing one or more transactions. It is secure, immutable and distributed. A distributed ledger.

What is Offchain Data? Any non-numeric, object, not aggregation based, supporting data that may have to be altered at a future date.

So what is stored in the Blockchain? Crypto signatures of previous and next blocks, timestamps, transactions and transaction data only. Pointers to offchain supporting data.

Translation Layer/Engine Block Offchain Identities Each peer in a blockchain will have a full copy of the blockchain Will only be responsible for storing their own offchain data. Can lead to questions of data security. Each DOC, BLOB, CLOB, JPEG or set of fields in a database record will have a hash signature in the transaction data with the blockchain Hash will be reverified each time an offchain data source is referenced. Blockchain Network built on Hyperledger Fabric IBM Cloud Translation Layer/Engine Smart Contract DAPP

Hash For a single object such as a picture, PDF or DOC file calculate the objects hash and compare it to the stored hash CREATE TABLE bfile_table ( Record_Hash_value RAW(256) NOT NULL, exfile_hash RAW(256) NOT NULL, Contents Varchar2(2000), file_type Varchar2(32), F_exfile BFILE); A simple table of the hash data and bfile pointer is all that is needed.

JPEG Hash Flow Chart Transaction requires offchain object Hash is generated for object and hash is stored with transaction Hash is placed in file manager/database Location pointer and object hash is stored in database Record is hashed and hash stored in database and transaction. Provides: Hash to verify object is same Hash to verify location is same Hash to verify record not changed JPEG Hash

Verification But what about multi-field records in a database CREATE TABLE customer ( custnum NUMBER(7) PRIMARY KEY, rec_hash RAW(256) NOT NULL, fname VARCHAR2(15) NOT NULL, lname VARCHAR2(24) NOT NULL, cc_num NUMBER(16) ENCRYPT, ver_code NUMBER(4), expdate DATE DEFAULT (sysdate), photo BLOB, street_no NUMBER(6), street_nam VARCHAR2(32), apt_num NUMBER(4), town VARCHAR2(32), city VARCHAR2(32), zip4 NUMBER(9) ) TABLESPACE customer_tbs STORAGE ( INITIAL 50K); But what about multi-field records in a database Names, addresses, SSN, dates of birth, credit card data All controlled by GDPR requirements Will be slowly changing over time Will have to be stored offchain How do you verify them?

Record field Verification Can do field by field Similar to the slowly changing type two dimension in data warehouses Can solve the problem similar to ways that are done in data warehouses

Record Field Validation Multiple text, date, object and number fields forces the blockchain manager to: Do difficult, field-by-field verification of the record use of some other form of field verification to determine if a field has changed in a record When a field is altered into a slowly changing record the old record is marked as invalid either by a flag or via a date field or set of date fields allowing for a bi-temporal record. Requires a new transaction be inserted into the blockchain indicating a source record has been altered and reverified by the certifying agent or agents.

Making Hash of Data Generate a hash for all the pertinent sections of the record, and store this hash with the blockchain record With each retrieval for each record that is to be verified, a hash is calculated for the important fields of the record. This new hash is compared to the stored hash in the blockchain transaction and if the signatures match, the record is used If the signature is changed, the new fields are marked as questionable, the old is marked as invalid and the new marked as the current record with a new transaction record stored in the block chain showing the change. If the new record matches none of the existing signatures, the record is inserted but no existing records are altered. Records marked as questionable must be reviewed by the certifying authority before they are used.

Making Hash of Data GDPR specifies that data personal information (PII) must be erasable. PII cannot be stored in the blockchain but must be stored in offchain storage. Think of an example of a customer table with a hash code inserted based on the important data stored in each row As an alternative, a hash based function based index could be used. The hash would is also stored in the blockchain and used to link to the record

Making Hash of Data CREATE TABLE customer ( custnum NUMBER(7) PRIMARY KEY, rec_hash RAW(256) NOT NULL, fname VARCHAR2(15) NOT NULL, lname VARCHAR2(24) NOT NULL, cc_num NUMBER(16) ENCRYPT, ver_code NUMBER(4), expdate DATE DEFAULT (sysdate), photo BLOB, street_no NUMBER(6), street_nam VARCHAR2(32), apt_num NUMBER(4), town VARCHAR2(32), city VARCHAR2(32), zip4 NUMBER(9) ) TABLESPACE customer_tbs STORAGE ( INITIAL 50K);

Making Hash of Data Hash and CRC codes can be difficult to code. Oracle provides DBMS_CRYPTO.HASH DBMS_CRYPTO can be used for Function based indexes In table columns Virtual columns Let’s look a quick examples

Using DBMS_CRYPTO.HASH Grant EXECUTE privilege on the DBMS_CRYPTO package to owner:   GRANT EXECUTE ON DBMS_CRYPTO TO BLCH; To use a function in INSERT statements it must be deterministic Since it generates a hash, you cannot call it with a null value, so a wrapper is required  CREATE OR REPLACE FUNCTION get_sign (clb clob) RETURN RAW DETERMINISTIC AS signature RAW(128); clb2 clob; BEGIN IF clb IS NULL THEN clb2:= to_clob('1'); ELSE clb2:=clb; END IF; Signature:=sys.DBMS_CRYPTO.HASH(clb2,4); RETURN signature; END; Note: The 4 in the call to HASH indicates to use a DES256 bit hash

Making Hash of Data Notice default value of '1' if the input value is null, because the function generates a hash signature you cannot pass it a null value Note if your offchain record contains a pointer to a bfile (externally stored object) you should store a hash for the bfile object for verifiction when retrieving that object. The record containing the pointer is hashed, guaranteeing no one can repoint the record to a different object and the object can be verified

Using Hash: Multiple Methods Generate a function based index. Eliminate the need to have an extra column Useful when using existing datastores as a source or target for blockchains Allows you to regenerate the index without having to update the complete table each time there may be columns added to the record Use it as an INSERT into a normal table column with an index Fastest Use it for a virtual column and index No physical mod to table

With and Without Function Based Index With FBI Elapsed: 00:00:00.01 Statistics -------------------------------------------------- 44 recursive calls 5 db block gets 11 consistent gets 0 physical reads 1052 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed Without FBI Elapsed: 00:00:04.67 Statistics --------------------------------------------------- 101 recursive calls 5 db block gets 1466 consistent gets 0 physical reads 0 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 5 sorts (memory) 0 sorts (disk) 1 rows processed Pros: Fast No changes to table Only requires index rebuild if column added Cons: Adds object to DB Must use query rewrite

Regular Table column With and Without index Without Index Elapsed: 00:00:02.28 Statistics ----------------------------------------------------- 22 recursive calls 0 db block gets 1698 consistent gets 0 physical reads 0 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed With Index Elapsed: 00:00:00.01 Statistics ----------------------------------------------------- 22 recursive calls 0 db block gets 8 consistent gets 0 physical reads 0 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed Cons: Adds object to DB Adds column to table Requires table update and index rebuild if column added Pros: Fast

Virtual Table Column With and Without index With Index Elapsed: Elapsed: 00:00:00.01 Statistics ----------------------------------------------------- 42 recursive calls 0 db block gets 10 consistent gets 0 physical reads 0 redo size 548 bytes sent via SQL*Net to client 607 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed Without Index Elapsed: 00:00:04.60 Statistics ----------------------------------------------------- 53 recursive calls 0 db block gets 1709 consistent gets 0 physical reads 0 redo size 548 bytes sent via SQL*Net to client 607 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 2 sorts (memory) 0 sorts (disk) 1 rows processed Pros: Fast Cons: Adds object to DB Adds column to table Requires table column drop and re-add and index rebuild if column added

All together now… With FBI Elapsed: 00:00:00.01 Statistics -------------------------------------------------- 44 recursive calls 5 db block gets 11 consistent gets 0 physical reads 1052 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed Regular Column With Index Elapsed: 00:00:00.01 Statistics ----------------------------------------------------- 22 recursive calls 0 db block gets 8 consistent gets 0 physical reads 0 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed Virtual Column With Index Elapsed: Elapsed: 00:00:00.01 Statistics ----------------------------------------------------- 42 recursive calls 0 db block gets 10 consistent gets 0 physical reads 0 redo size 548 bytes sent via SQL*Net to client 607 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed Overall a normal column, populated at row creation, with an index, performed best. Fewer recursive calls Fewer consistent gets No redo generation

This was limited testing Only 72,600 rows on disk Performance could be impacted if it was millions of records because of disk IBM FlashSystem mitigates this

IBM Systems Flash Storage Offerings Portfolio Enhanced data storage functions, economics and flexibility with sophisticated virtualization NVMe end-to-end FlashSystem A9000 A9000R High End Enterprise FlashSystem 9100 Models 9110, 9150 Enterprise Class, NVMe accelerated Multi-Cloud Enabled Cloud Service Providers Storwize V5030F / V7000F Entry / Mid-Range SVC Scale-out clustering Simplified management Flexible consumption model Virtualized, flash-optimized, modular storage Enterprise heterogeneous data services and selectable data reduction IBM Elastic Storage Server Big Data Consolidate file & object workloads Faster data analysis Global sharing DS888xF Business Critical z/OS / AIX Power HA Power i HA Business critical, deepest integration with z Systems Superior performance and reliability Three-site / Four-site replication DS8882F, DS8888F, DS8886F, DS8884F IBM Power Systems NVMe FlashCore Module Superior endurance & better performance FIPS 140-2 Hardware Compression IBM FlashCore™ Technology Optimized FlashSystem 900 Application acceleration Extreme performance Targeting database acceleration Large Grid scale Full time data reduction As you all know, FlashSystem 9100 is the latest offering in our comprehensive portfolio of All-Flash Arrays, so how do you position one vs the other. For those familiar with a previous generation of this chart, what you will notice, is that FlashSystem V9000 is no longer on here. We are not withdrawing V9000 from market at this point, and so clients who have already invested in those can continue to add to their estate. So lets start on the left. You can see we have a common set of values with our Spectrum Virtualize based offerings, being Storwize ‘F’ models for entry to mid-range all-flash requirements and the new NVMe end-to-end offering, the FlashSystem 9100. They offer scale-out clustering, simplified management, a flexible consumption model, storage virtualization, a comprehensive set of enterprise data services and optional data reduction capability. We have FlashSystem 9100 and FlashSystem A9000 leveraging our FlashCore technology, with 9100 obviously using the NVMe FlashCore Module. A9000 has a common set of values with the A9000R, leveraging the Spectrum Accelerate code to deliver simplified management, large grid scale and full-time data reduction, and in common with the Spectrum Virtualize offerings, has a flexible consumption model. For clients with file and object workload requirements, we have the Elastic Storage Server based on the Spectrum Scale code. And last but not least, the DS8000 range for business critical, mainframe and multi-site replication requirements. Buts lets look at a bit more detail and workload to help with positioning these.

Summary Overall a normal column, populated at row creation, with an index, performed best. Fewer recursive calls Fewer consistent gets No redo generation In a stable environment with no changes to table structure, a normal column and index might be best from a database point of view Would require the most application changes In an environment where changes to existing tables are forbidden, then the FBI is the best choice No table changes Least impact to application For the least required programming changes to existing applications, the FBI and virtual column with index would be best. The virtual columns performance, while slightly better than the FBI, also requires the change to a table and addition of an index.

#JourneyToMulticloud

Detailed Examples: Creating the Test Table

Examples: Creating the Test table First create a test table A simple CTAS against a large data dictionary table such as DBA_OBJECTS will work SQL> CREATE TABLE test AS SELECT * FROM dba_objects;   Table created.

Examples: Creating the Test Table SQL> desc test Name Null? Type ----------------------------------------- -------- ------------------- OWNER VARCHAR2(128) OBJECT_NAME VARCHAR2(128) SUBOBJECT_NAME VARCHAR2(128) OBJECT_ID NUMBER … MODIFIED_APPID NUMBER MODIFIED_VSNID NUMBER   SQL> SELECT COUNT(*) FROM test; COUNT(*) ---------- 72600

Examples: Creating the Test Table SQL> CREATE TABLE test2 AS SELECT * FROM dba_objects;   SQL> SELECT COUNT(*) FROM test2; COUNT(*) ---------- 72601 SQL> DROP test; SQL> RENAME test 2 to test; This was to allow me to use selects against TEST in the table TEST, if you just use an existing table, no need for this

Detailed Examples: Using a Function Based Index

Using a Function based Index SQL> CREATE INDEX test_fbi ON test(get_sign(owner||object_name||subobject_name||to_char(object_id));  Index created. In order to have the FBI used, you must have query rewrite enabled and the table analyzed, in Oracle12/18 it is by default, Verify: query_rewrite_enabled = true query_rewrite_integrity = trusted or enforced SQL> show parameter query_ NAME TYPE VALUE ------------------------------------ ----------- ----------------------- query_rewrite_enabled string TRUE query_rewrite_integrity string enforced SQL> ANALYZE TABLE test COMPUTE STATISTICS;  Table analyzed.

Using a Function Based Index Turn on statistics and tracing and test: SQL> set autotrace on   SQL> set timing on SQL> SELECT a.object_type FROM test a 2 WHERE 3 get_sign('SYSTEM'||'TEST'||''||to_char(73840)) = get_sign(a.owner||a.object_name||a.subobject_name||to_char(object_id)); Elapsed: 00:00:00.01 OBJECT_TYPE ---------- TABLE

Using a Function Based Index Execution Plan ---------------------------------------------------------- Plan hash value: 1589953258 ---------------------------------------------------------------------- | Id | Operation | Name |Rows | Bytes |Cost (%CPU)| Time | ----------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1| 36 | 3(0)| 00:00:01| | 1 | TABLE ACCESS BY INDEX ROWID BATCHED| TEST | 1| 36 | 3(0)| 00:00:01| |* 2 | INDEX RANGE SCAN |TEST_FBI| 1 | | 1(0)| 00:00:01| Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("SYSTEM"."GET_SIGN"("OWNER"||"OBJECT_NAME"||"SUBOBJECT_NAME||TO_CHAR(73840)")="GET _SIGN(‘SYSTEMTEST73840’)) Statistics 44 recursive calls 5 db block gets 11 consistent gets 0 physical reads 1052 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed

Without the FBI SQL> DROP INDEX test_fbi; Index dropped.   Index dropped. SQL> SELECT a.object_type FROM test a 2 WHERE 3 get_sign('SYSTEM'||'TEST’||TO_CHAR(73840)'') = get_sign(a.owner||a.object_name||a.subobject_name||to_char(a.object_id)); Elapsed: 00:00:04.67 OBJECT_TYPE ----------- TABLE

Without the FBI Execution Plan ---------------------------------------------------------- Plan hash value: 1357081020 ----------------------------------------------------------------------- | Id | Operation | Name | Rows| Bytes |Cost (%CPU)| Time | | 0 | SELECT STATEMENT | | 726 | 31944 | 404 (5)| 00:00:01 | |* 1 | TABLE ACCESS FULL| TEST | 726 | 31944 | 404 (5)| 00:00:01 | Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter("GET_SIGN"("A"."OWNER"||"A"."OBJECT_NAME"||"A"."SUBOBJECT||TO_CHAR(A>OBJECT_ID)_ NAME")="GET_SIGN"('SYSTEMTEST73840')) Statistics 101 recursive calls 5 db block gets 1466 consistent gets 0 physical reads 0 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 5 sorts (memory) 0 sorts (disk) 1 rows processed

Detailed Examples: Normal Column Plus Index

Normal Table Column Plus Index SQL> alter table test add signature raw(128);   Table altered. SQL> update test a set a.signature=get_sign(a.owner||a.object_name||a.subobject_name||to_char(a.object_id)); 72609 rows updated. SQL> commit; Commit complete.

Normal Table Column No Index First, with no index: SQL> SELECT a.object_type FROM test a 2 WHERE 3 get_sign('SYSTEM'||'TEST'||''||TO_CHAR(73840)) = a.signature;   OBJECT_TYPE ------------ TABLE Elapsed: 00:00:02.28 Better than doing a column by column conversion, but not as good as a FBI

Normal Table Column no Index Execution Plan ---------------------------------------------------------- Plan hash value: 1357081020 ----------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |Cost (%CPU)| Time | | 0 | SELECT STATEMENT | | 726 | 97284 | 397 (4)| 00:00:01| |* 1 | TABLE ACCESS FULL| TEST | 726 | 97284 | 397 (4)| 00:00:01| Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter("A"."SIGNATURE"="GET_SIGN"('SYSTEMTEST73840')) Statistics 22 recursive calls 0 db block gets 1698 consistent gets 0 physical reads 0 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed

Normal Table Column Plus Index (Add index) SQL> create index test_ui on test(signature);  Index created. SQL> analyze table test compute statistics;  Table analyzed.   SQL> SELECT a.object_type FROM test a 2 WHERE 3 get_sign('SYSTEM'||'TEST'||’’||TO_CHAR(73840)’) = a.signature; OBJECT_TYPE ----------- TABLE Elapsed: 00:00:00.01 With the index the performance is on par with the FBI

Normal Table Column Plus Index Execution Plan ---------------------------------------------------------- Plan hash value: 1530865433 ---------------------------------------------------------------------- | Id | Operation | Name | Rows|Bytes|Cost(%CPU| Time| ------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1| 36| 3(0)| 00:00:01| | 1 | TABLE ACCESS BY INDEX ROWID BATCHED|TEST | 1| 36| 3(0)| 00:00:01| |* 2 | INDEX RANGE SCAN |TEST_UI| 1| | 1(0)| 00:00:01| ------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("A"."SIGNATURE"="GET_SIGN"('SYSTEMTEST73840')) Statistics 22 recursive calls 0 db block gets 8 consistent gets 0 physical reads 0 redo size 545 bytes sent via SQL*Net to client 608 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed

Detailed Examples: Virtual Table Column

Table Virtual Column with Index A virtual column calculates values based on other columns automatically SQL> alter table test drop column signature; -- Drop signature column, also drops its index Table altered. SQL> alter table test add hash RAW(2000) GENERATED ALWAYS AS (get_sign(owner||object_name||subobject_name||to_char(object_id))) VIRTUAL; -- add our virtual column, automatically populates SQL> create index test_ind on test(hash); -- Add our index Index created.   SQL> ANALYZE TABLE test COMPUTE STATISTICS SQL> SELECT a.object_type FROM test a 2 WHERE 3* get_sign('SYSTEM'||'TEST'||''||to_char(73840)) = hash; OBJECT_TYPE ----------------------- TABLE  Elapsed: 00:00:00.01

Table Virtual Column with Index Execution Plan ---------------------------------------------------------- Plan hash value: 1744442303 -------------------------------------------------------------------------------------- | Id | Operation | Name |Rows |Bytes|Cost (%CPU)| Time | | 0 | SELECT STATEMENT | |72609|9785K| 1 (0)| 00:00:01 | | 1 | TABLE ACCESS BY INDEX ROWID BATCHED| TEST |72609|9785K| 1 (0)| 00:00:01 | |* 2 | INDEX RANGE SCAN | TEST_IND| 1| | 1 (0)| 00:00:01 | Predicate Information (identified by operation id): --------------------------------------------------- 2 - access("HASH"="GET_SIGN"('SYSTEMTEST73840')) Statistics 42 recursive calls 0 db block gets 10 consistent gets 0 physical reads 0 redo size 548 bytes sent via SQL*Net to client 607 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 0 sorts (memory) 0 sorts (disk) 1 rows processed

Table Virtual Column without Index SQL> DROP INDEX test_ind; Index dropped SQL> SELECT a.object_type FROM test a 2 WHERE 3* get_sign('SYSTEM'||'TEST'||''||to_char(73850)) = a.hash SQL> /   OBJECT_TYPE ----------------------- TABLE Elapsed: 00:00:04.60

Table Virtual Column without Index Execution Plan ---------------------------------------------------------- Plan hash value: 1357081020 -------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | | 0 | SELECT STATEMENT | | 726 | 97K| 404 (5)| 00:00:01 | |* 1 | TABLE ACCESS FULL| TEST | 726 | 97K| 404 (5)| 00:00:01 | Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter("A"."HASH"="GET_SIGN"('SYSTEMTEST73850')) Statistics 53 recursive calls 0 db block gets 1709 consistent gets 0 physical reads 0 redo size 548 bytes sent via SQL*Net to client 607 bytes received via SQL*Net from client 2 SQL*Net roundtrips to/from client 2 sorts (memory) 0 sorts (disk) 1 rows processed