1. Working on the Chain Gang
Utilizing Oracle for Offchain data storage
Eric Herzog
CMO
Vice President, Worldwide Storage Channels
IBM Storage Systems and Software-Defined Infrastructure
@zoginstor

2. Introduction
Blockchain’s relationship to offchain storage is similar to fact and dimension tables in a star schema
The Blockchain’s is the Fact table and the offchain storage the dimension tables

3. Introduction
In a Data Warehouse the Fact table stores a row of foreign keys pointing to the dimension tables and the “facts” that result from that intersection of data sources.
The blockchain transaction should only store the needed hash IDs, transaction codes and intersection data such as sums, amounts and aggregations resulting from the transaction
Offchain data becomes the dimension tables, storing all detail data

4. Just in case…
A chronologically ordered, cryptographically signature linked series of blocks containing one or more transactions. It is secure, immutable and distributed. A distributed ledger.

5. What is Offchain Data?
Any non-numeric, object, not aggregation based, supporting data that may have to be altered at a future date.

6. So what is stored in the Blockchain?
Crypto signatures of previous and next blocks, timestamps, transactions and transaction data only.
Pointers to offchain supporting data.

7. Translation Layer/Engine
Block Offchain Identities
Each peer in a blockchain will have a full copy of the blockchain
Will only be responsible for storing their own offchain data.
Can lead to questions of data security.
Each DOC, BLOB, CLOB, JPEG or set of fields in a database record will have a hash signature in the transaction data with the blockchain
Hash will be reverified each time an offchain data source is referenced.
Blockchain Network
built on Hyperledger Fabric
IBM Cloud
Translation Layer/Engine
Smart
Contract
DAPP

8. Hash
For a single object such as a picture, PDF or DOC file calculate the objects hash and compare it to the stored hash
CREATE TABLE bfile_table (
Record_Hash_value RAW(256) NOT NULL,
exfile_hash RAW(256) NOT NULL,
Contents Varchar2(2000),
file_type Varchar2(32),
F_exfile BFILE);
A simple table of the hash data and bfile pointer is all that is needed.

9. JPEG Hash Flow Chart Transaction requires offchain object
Hash is generated for object and hash is stored with transaction
Hash is placed in file manager/database
Location pointer and object hash is stored in database
Record is hashed and hash stored in database and transaction.
Provides:
Hash to verify object is same
Hash to verify location is same
Hash to verify record not changed
JPEG
Hash

10. Verification But what about multi-field records in a database
CREATE TABLE customer (
custnum NUMBER(7) PRIMARY KEY,
rec_hash RAW(256) NOT NULL,
fname VARCHAR2(15) NOT NULL,
lname VARCHAR2(24) NOT NULL,
cc_num NUMBER(16) ENCRYPT,
ver_code NUMBER(4),
expdate DATE DEFAULT (sysdate),
photo BLOB,
street_no NUMBER(6),
street_nam VARCHAR2(32),
apt_num NUMBER(4),
town VARCHAR2(32),
city VARCHAR2(32),
zip NUMBER(9) )
TABLESPACE customer_tbs
STORAGE ( INITIAL 50K);
But what about multi-field records in a database
Names, addresses, SSN, dates of birth, credit card data
All controlled by GDPR requirements
Will be slowly changing over time
Will have to be stored offchain
How do you verify them?

11. Record field Verification
Can do field by field
Similar to the slowly changing type two dimension in data warehouses
Can solve the problem similar to ways that are done in data warehouses

12. Record Field Validation
Multiple text, date, object and number fields forces the blockchain manager to:
Do difficult, field-by-field verification of the record
use of some other form of field verification to determine if a field has changed in a record
When a field is altered into a slowly changing record the old record is marked as invalid either by a flag or via a date field or set of date fields allowing for a bi-temporal record.
Requires a new transaction be inserted into the blockchain indicating a source record has been altered and reverified by the certifying agent or agents.

13. Making Hash of Data
Generate a hash for all the pertinent sections of the record, and store this hash with the blockchain record
With each retrieval for each record that is to be verified, a hash is calculated for the important fields of the record.
This new hash is compared to the stored hash in the blockchain transaction and if the signatures match, the record is used
If the signature is changed, the new fields are marked as questionable, the old is marked as invalid and the new marked as the current record with a new transaction record stored in the block chain showing the change.
If the new record matches none of the existing signatures, the record is inserted but no existing records are altered.
Records marked as questionable must be reviewed by the certifying authority before they are used.

14. Making Hash of Data
GDPR specifies that data personal information (PII) must be erasable.
PII cannot be stored in the blockchain but must be stored in offchain storage.
Think of an example of a customer table with a hash code inserted based on the important data stored in each row
As an alternative, a hash based function based index could be used.
The hash would is also stored in the blockchain and used to link to the record

15. Making Hash of Data CREATE TABLE customer (
custnum NUMBER(7) PRIMARY KEY,
rec_hash RAW(256) NOT NULL,
fname VARCHAR2(15) NOT NULL,
lname VARCHAR2(24) NOT NULL,
cc_num NUMBER(16) ENCRYPT,
ver_code NUMBER(4),
expdate DATE DEFAULT (sysdate),
photo BLOB,
street_no NUMBER(6),
street_nam VARCHAR2(32),
apt_num NUMBER(4),
town VARCHAR2(32),
city VARCHAR2(32),
zip NUMBER(9)
) TABLESPACE customer_tbs STORAGE ( INITIAL 50K);

16. Making Hash of Data Hash and CRC codes can be difficult to code.
Oracle provides DBMS_CRYPTO.HASH
DBMS_CRYPTO can be used for
Function based indexes
In table columns
Virtual columns
Let’s look a quick examples

17. Using DBMS_CRYPTO.HASH
Grant EXECUTE privilege on the DBMS_CRYPTO package to owner:
GRANT EXECUTE ON DBMS_CRYPTO TO BLCH;
To use a function in INSERT statements it must be deterministic
Since it generates a hash, you cannot call it with a null value, so a wrapper is required
 CREATE OR REPLACE FUNCTION get_sign (clb clob)
RETURN RAW
DETERMINISTIC AS
signature RAW(128);
clb2 clob;
BEGIN
IF clb IS NULL THEN
clb2:= to_clob('1');
ELSE
clb2:=clb;
END IF;
Signature:=sys.DBMS_CRYPTO.HASH(clb2,4);
RETURN signature;
END;
Note: The 4 in the call to HASH indicates to use a DES256 bit hash

18. Making Hash of Data
Notice default value of '1' if the input value is null, because the function generates a hash signature you cannot pass it a null value
Note if your offchain record contains a pointer to a bfile (externally stored object) you should store a hash for the bfile object for verifiction when retrieving that object.
The record containing the pointer is hashed, guaranteeing no one can repoint the record to a different object and the object can be verified

19. Using Hash: Multiple Methods
Generate a function based index.
Eliminate the need to have an extra column
Useful when using existing datastores as a source or target for blockchains
Allows you to regenerate the index without having to update the complete table each time there may be columns added to the record
Use it as an INSERT into a normal table column with an index
Fastest
Use it for a virtual column and index
No physical mod to table

20. With and Without Function Based Index
With FBI Elapsed: 00:00:00.01
Statistics
44 recursive calls
5 db block gets
11 consistent gets
0 physical reads
1052 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed
Without FBI Elapsed: 00:00:04.67
Statistics
101 recursive calls
5 db block gets
1466 consistent gets
0 physical reads
0 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
5 sorts (memory)
0 sorts (disk)
1 rows processed
Pros:
Fast
No changes to table
Only requires index rebuild if column added
Cons:
Adds object to DB
Must use query rewrite

21. Regular Table column With and Without index
Without Index Elapsed: 00:00:02.28
Statistics
22 recursive calls
0 db block gets
1698 consistent gets
0 physical reads
0 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed
With Index Elapsed: 00:00:00.01
Statistics
22 recursive calls
0 db block gets
8 consistent gets
0 physical reads
0 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed
Cons:
Adds object to DB
Adds column to table
Requires table update and index rebuild if column added
Pros:
Fast

22. Virtual Table Column With and Without index
With Index Elapsed: Elapsed: 00:00:00.01
Statistics
42 recursive calls
0 db block gets
10 consistent gets
0 physical reads
0 redo size
548 bytes sent via SQL*Net to client
607 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed
Without Index Elapsed: 00:00:04.60
Statistics
53 recursive calls
0 db block gets
1709 consistent gets
0 physical reads
0 redo size
548 bytes sent via SQL*Net to client
607 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
2 sorts (memory)
0 sorts (disk)
1 rows processed
Pros:
Fast
Cons:
Adds object to DB
Adds column to table
Requires table column drop and re-add and index rebuild if column added

23. All together now…
With FBI Elapsed: 00:00:00.01
Statistics
44 recursive calls
5 db block gets
11 consistent gets
0 physical reads
1052 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed
Regular Column With Index Elapsed: 00:00:00.01
Statistics
22 recursive calls
0 db block gets
8 consistent gets
0 physical reads
0 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed
Virtual Column With Index Elapsed: Elapsed: 00:00:00.01
Statistics
42 recursive calls
0 db block gets
10 consistent gets
0 physical reads
0 redo size
548 bytes sent via SQL*Net to client
607 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed
Overall a normal column, populated at row creation, with an index, performed best.
Fewer recursive calls
Fewer consistent gets
No redo generation

24. This was limited testing
Only 72,600 rows on disk
Performance could be impacted if it was millions of records because of disk
IBM FlashSystem mitigates this

25. IBM Systems Flash Storage Offerings Portfolio
Enhanced data storage functions,
economics and flexibility with
sophisticated virtualization
NVMe end-to-end
FlashSystem
A9000
A9000R
High End Enterprise
FlashSystem 9100
Models 9110, 9150
Enterprise Class,
NVMe accelerated
Multi-Cloud Enabled
Cloud Service Providers
Storwize
V5030F / V7000F
Entry / Mid-Range
SVC
Scale-out clustering
Simplified management
Flexible consumption model
Virtualized, flash-optimized, modular storage
Enterprise heterogeneous data services and selectable data reduction
IBM Elastic Storage Server
Big Data
Consolidate file & object workloads
Faster data analysis
Global sharing
DS888xF
Business Critical
z/OS / AIX
Power HA
Power i HA
Business critical, deepest integration with z Systems
Superior performance and reliability
Three-site / Four-site replication
DS8882F, DS8888F, DS8886F, DS8884F
IBM Power
Systems
NVMe FlashCore Module
Superior endurance & better performance
FIPS 140-2
Hardware Compression
IBM FlashCore™ Technology Optimized
FlashSystem 900
Application acceleration
Extreme performance
Targeting database acceleration
Large Grid scale
Full time data reduction
As you all know, FlashSystem 9100 is the latest offering in our comprehensive portfolio of All-Flash Arrays, so how do you position one vs the other.
For those familiar with a previous generation of this chart, what you will notice, is that FlashSystem V9000 is no longer on here. We are not withdrawing V9000 from market at this point, and so clients who have already invested in those can continue to add to their estate.
So lets start on the left. You can see we have a common set of values with our Spectrum Virtualize based offerings, being Storwize ‘F’ models for entry to mid-range all-flash requirements and the new NVMe end-to-end offering, the FlashSystem They offer scale-out clustering, simplified management, a flexible consumption model, storage virtualization, a comprehensive set of enterprise data services and optional data reduction capability.
We have FlashSystem 9100 and FlashSystem A9000 leveraging our FlashCore technology, with 9100 obviously using the NVMe FlashCore Module.
A9000 has a common set of values with the A9000R, leveraging the Spectrum Accelerate code to deliver simplified management, large grid scale and full-time data reduction, and in common with the Spectrum Virtualize offerings, has a flexible consumption model.
For clients with file and object workload requirements, we have the Elastic Storage Server based on the Spectrum Scale code. And last but not least, the DS8000 range for business critical, mainframe and multi-site replication requirements. Buts lets look at a bit more detail and workload to help with positioning these.

26. Summary
Overall a normal column, populated at row creation, with an index, performed best.
Fewer recursive calls
Fewer consistent gets
No redo generation
In a stable environment with no changes to table structure, a normal column and index might be best from a database point of view
Would require the most application changes
In an environment where changes to existing tables are forbidden, then the FBI is the best choice
No table changes
Least impact to application
For the least required programming changes to existing applications, the FBI and virtual column with index would be best.
The virtual columns performance, while slightly better than the FBI, also requires the change to a table and addition of an index.

27. #JourneyToMulticloud

28. Detailed Examples: Creating the Test Table

29. Examples: Creating the Test table
First create a test table
A simple CTAS against a large data dictionary table such as DBA_OBJECTS will work
SQL> CREATE TABLE test AS SELECT * FROM dba_objects;
Table created.

30. Examples: Creating the Test Table
SQL> desc test
Name Null? Type
OWNER VARCHAR2(128)
OBJECT_NAME VARCHAR2(128)
SUBOBJECT_NAME VARCHAR2(128)
OBJECT_ID NUMBER …
MODIFIED_APPID NUMBER
MODIFIED_VSNID NUMBER
SQL> SELECT COUNT(*) FROM test;
COUNT(*)
72600

31. Examples: Creating the Test Table
SQL> CREATE TABLE test2 AS SELECT * FROM dba_objects;
SQL> SELECT COUNT(*) FROM test2;
COUNT(*)
72601
SQL> DROP test;
SQL> RENAME test 2 to test;
This was to allow me to use selects against TEST in the table TEST, if you just use an existing table, no need for this

32. Detailed Examples: Using a Function Based Index

33. Using a Function based Index
SQL> CREATE INDEX test_fbi ON test(get_sign(owner||object_name||subobject_name||to_char(object_id));
 Index created.
In order to have the FBI used, you must have query rewrite enabled and the table analyzed, in Oracle12/18 it is by default, Verify:
query_rewrite_enabled = true
query_rewrite_integrity = trusted or enforced
SQL> show parameter query_
NAME TYPE VALUE
query_rewrite_enabled string TRUE
query_rewrite_integrity string enforced
SQL> ANALYZE TABLE test COMPUTE STATISTICS;
 Table analyzed.

34. Using a Function Based Index
Turn on statistics and tracing and test:
SQL> set autotrace on
SQL> set timing on
SQL> SELECT a.object_type FROM test a
2 WHERE
3 get_sign('SYSTEM'||'TEST'||''||to_char(73840)) = get_sign(a.owner||a.object_name||a.subobject_name||to_char(object_id));
Elapsed: 00:00:00.01
OBJECT_TYPE
TABLE

35. Using a Function Based Index
Execution Plan
Plan hash value:
| Id | Operation | Name |Rows | Bytes |Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 1| | 3(0)| 00:00:01|
| 1 | TABLE ACCESS BY INDEX ROWID BATCHED| TEST | 1| | 3(0)| 00:00:01|
|* 2 | INDEX RANGE SCAN |TEST_FBI| 1 | | 1(0)| 00:00:01|
Predicate Information (identified by operation id):
2 - access("SYSTEM"."GET_SIGN"("OWNER"||"OBJECT_NAME"||"SUBOBJECT_NAME||TO_CHAR(73840)")="GET
_SIGN(‘SYSTEMTEST73840’))
Statistics
44 recursive calls
5 db block gets
11 consistent gets
0 physical reads
1052 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed

36. Without the FBI SQL> DROP INDEX test_fbi; Index dropped.
Index dropped.
SQL> SELECT a.object_type FROM test a
2 WHERE
3 get_sign('SYSTEM'||'TEST’||TO_CHAR(73840)'') = get_sign(a.owner||a.object_name||a.subobject_name||to_char(a.object_id));
Elapsed: 00:00:04.67
OBJECT_TYPE
TABLE

37. Without the FBI Execution Plan
Plan hash value:
| Id | Operation | Name | Rows| Bytes |Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 726 | | (5)| 00:00:01 |
|* 1 | TABLE ACCESS FULL| TEST | 726 | | (5)| 00:00:01 |
Predicate Information (identified by operation id):
1 - filter("GET_SIGN"("A"."OWNER"||"A"."OBJECT_NAME"||"A"."SUBOBJECT||TO_CHAR(A>OBJECT_ID)_
NAME")="GET_SIGN"('SYSTEMTEST73840'))
Statistics
101 recursive calls
5 db block gets
1466 consistent gets
0 physical reads
0 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
5 sorts (memory)
0 sorts (disk)
1 rows processed

38. Detailed Examples:
Normal Column Plus Index

39. Normal Table Column Plus Index
SQL> alter table test add signature raw(128);
Table altered.
SQL> update test a set a.signature=get_sign(a.owner||a.object_name||a.subobject_name||to_char(a.object_id));
72609 rows updated.
SQL> commit;
Commit complete.

40. Normal Table Column No Index
First, with no index:
SQL> SELECT a.object_type FROM test a
2 WHERE
3 get_sign('SYSTEM'||'TEST'||''||TO_CHAR(73840)) = a.signature;
OBJECT_TYPE
TABLE
Elapsed: 00:00:02.28
Better than doing a column by column conversion, but not as good as a FBI

41. Normal Table Column no Index
Execution Plan
Plan hash value:
| Id | Operation | Name | Rows | Bytes |Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | 726 | | (4)| 00:00:01|
|* 1 | TABLE ACCESS FULL| TEST | 726 | | (4)| 00:00:01|
Predicate Information (identified by operation id):
1 - filter("A"."SIGNATURE"="GET_SIGN"('SYSTEMTEST73840'))
Statistics
22 recursive calls
0 db block gets
1698 consistent gets
0 physical reads
0 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed

42. Normal Table Column Plus Index (Add index)
SQL> create index test_ui on test(signature);
 Index created.
SQL> analyze table test compute statistics;
 Table analyzed.
SQL> SELECT a.object_type FROM test a
2 WHERE
3 get_sign('SYSTEM'||'TEST'||’’||TO_CHAR(73840)’) = a.signature;
OBJECT_TYPE
TABLE
Elapsed: 00:00:00.01
With the index the performance is on par with the FBI

43. Normal Table Column Plus Index
Execution Plan
Plan hash value:
| Id | Operation | Name | Rows|Bytes|Cost(%CPU| Time|
| 0 | SELECT STATEMENT | | 1| 36| 3(0)| 00:00:01|
| 1 | TABLE ACCESS BY INDEX ROWID BATCHED|TEST | 1| 36| 3(0)| 00:00:01|
|* 2 | INDEX RANGE SCAN |TEST_UI| 1| | 1(0)| 00:00:01|
Predicate Information (identified by operation id):
2 - access("A"."SIGNATURE"="GET_SIGN"('SYSTEMTEST73840'))
Statistics
22 recursive calls
0 db block gets
8 consistent gets
0 physical reads
0 redo size
545 bytes sent via SQL*Net to client
608 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed

44. Detailed Examples:
Virtual Table Column

45. Table Virtual Column with Index
A virtual column calculates values based on other columns automatically
SQL> alter table test drop column signature; -- Drop signature column, also drops its index
Table altered.
SQL> alter table test add hash RAW(2000) GENERATED ALWAYS AS (get_sign(owner||object_name||subobject_name||to_char(object_id))) VIRTUAL; -- add our virtual column, automatically populates
SQL> create index test_ind on test(hash); -- Add our index
Index created.
SQL> ANALYZE TABLE test COMPUTE STATISTICS
SQL> SELECT a.object_type FROM test a
2 WHERE
3* get_sign('SYSTEM'||'TEST'||''||to_char(73840)) = hash;
OBJECT_TYPE
TABLE
 Elapsed: 00:00:00.01

46. Table Virtual Column with Index
Execution Plan
Plan hash value:
| Id | Operation | Name |Rows |Bytes|Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | |72609|9785K| 1 (0)| 00:00:01 |
| 1 | TABLE ACCESS BY INDEX ROWID BATCHED| TEST |72609|9785K| 1 (0)| 00:00:01 |
|* 2 | INDEX RANGE SCAN | TEST_IND| 1| | 1 (0)| 00:00:01 |
Predicate Information (identified by operation id):
2 - access("HASH"="GET_SIGN"('SYSTEMTEST73840'))
Statistics
42 recursive calls
0 db block gets
10 consistent gets
0 physical reads
0 redo size
548 bytes sent via SQL*Net to client
607 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed

47. Table Virtual Column without Index
SQL> DROP INDEX test_ind;
Index dropped
SQL> SELECT a.object_type FROM test a
2 WHERE
3* get_sign('SYSTEM'||'TEST'||''||to_char(73850)) = a.hash
SQL> /
OBJECT_TYPE
TABLE
Elapsed: 00:00:04.60

48. Table Virtual Column without Index
Execution Plan
Plan hash value:
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
| 0 | SELECT STATEMENT | | | 97K| (5)| 00:00:01 |
|* 1 | TABLE ACCESS FULL| TEST | | 97K| (5)| 00:00:01 |
Predicate Information (identified by operation id):
1 - filter("A"."HASH"="GET_SIGN"('SYSTEMTEST73850'))
Statistics
53 recursive calls
0 db block gets
1709 consistent gets
0 physical reads
0 redo size
548 bytes sent via SQL*Net to client
607 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
2 sorts (memory)
0 sorts (disk)
1 rows processed