History Database - Typical historical query requirements show history of values (states) for assetX show full transaction history for assetX, for example to understand what other assets were in the same transactions show assets that have been owned by Alice show assets that were at locationY show values/ownership at a certain point in time Simple provenance queries can be implemented in any key/value (LevelDB) or document datastore (CouchDB) Requires a datastore that supports rich query (e.g. CouchDB)

History Database - Typical historical query requirements Priority for v1 show history of values (states) for assetX show full transaction history for assetX, for example to understand what other assets were in the same transactions show assets that have been owned by Alice show assets that were at locationY show values/ownership at a certain point in time Simple provenance queries can be implemented in any key/value (LevelDB) or document datastore (CouchDB) Requires a datastore that supports rich query (e.g. CouchDB)

History of a Key - GetTransactionsForKey() API GetTransactionsForKey() API uses history database to return history for a key. Can be used for simple provenance scenarios. Expose to SDK and chaincode (similar to ExecuteQuery) Option to return historical values (states) or historical transaction details for a key.

History of a Key - GetTransactionsForKey() API Determining history (simple provenance) of a key is supported by API GetTransactionsForKey(). This API could be called from SDK or from within chaincode. To support this API, there will be an optional history database with a record for each historic key value. The history database is actually an ‘index’ or ‘materialized view’ of the blockchain, to handle historical queries more efficiently than scanning the entire blockchain data set. Some clients will want to know the history of key values only, other clients will want to know the full transaction context for each historic key value. Therefore the API will return a list of key modifications that includes summary information (txid, timestamp) and an option to return the value or full transaction details associated with the historical modification. Note: While the value itself can be found within the transaction details WriteSet, providing an option to return the value directly will be simpler for those applications that only need the history of values. Since this will be exposed to chaincode applications, this simplification is important. // GetTransactionsForKey returns an iterator that contains all the transactions that modified the given key. GetTransactionsForKey(namespace string, key string, includeValues bool, includeTransactions bool) (ResultsIterator, error) The returned ResultsIterator contains results of type KeyModification: type KeyModification struct{ Txid string TxTimestamp time.Time Value []byte Transaction ?.Transaction } Note: will likely need ordering service to provide a system timestamp at block level. All the transactions within the block would inherit the same timestamp. Optional in response. Optional in response. Note: I think we need to author a Transaction struct that is friendly for chaincode applications.

Implementation – Common approach for simple provenance History database is enabled via a config setting in core.yaml. In v1, store history records in LevelDB, regardless of state database choice. There is a record inserted into history database for each key value written to state database. Key for this record will be: namespace~key~blocknum~trannum (‘namespace’ represents chaincode name. ‘~’ represents null character.) (blocknum,trannum) ensures a unique key for each historical value. Example historical keys for a chaincode key that has been updated twice, at (block 123, tran 12) and (block 456, tran 7): chaincode1~key1~0000000000123~000000000012 chaincode1~key1~0000000000456~000000000007 GetTransactionsForKey() for (chaincode1,key1) would do a key range query on chaincode1~key1 to pick up all chaincode1~key1 records. Results will indicate the set of (blocknum,trannum) transactions that updated this key In the example above, that would be (block 123, tran 12) and (block 456, tran 7). Finally, build the API response by using the (blocknum,trannum) to pull the value and/or transaction details from blockchain block storage. There is no need to duplicate this info in the history database for simple provenance on goleveldb (historic key values can be null in history database) Note, a new block storage index will map (blocknum,trannum) to the file storage location for this block tran, similar in concept to the other block storage indexes: blocknum~trannum  Transaction SegNo + offset

History Database – CouchDB extensions (future) To support rich queries against historical data, when using queryable database such as CouchDB, in addition to storing the historic key updates, also store the key values at that point in time. For example: chaincode1~key1~0000000000123~000000000012  Value at block123, tran12 chaincode1~key1~0000000000456~000000000007  Value to block456, tran7 Storing the historic key values will support rich queries against the data history, for example using a new API such as ExecuteQueryHistory(queryString string). Example rich queries: show assets that were at locationY show assets that have been owned by Alice Also, with a copy of the value in history database, if GetTransactionsForKey() has includeValues only, then we can return the values from the CouchDB history database rather than looking up the values from block storage. CouchDB will not have the txid to return, potentially we will add that to the CouchDB data wrapper, in addition to version, so that it will be available to return efficiently.

History Database - Other details History database is separate from state database, to provide isolation in terms of resources, performance, scalability, pluggability, etc Same guarantees of consistency as state database Use savepoints to ensure consistency with blockchain block storage Peer startup checks savepoints to ensure consistency after peer crash Automatically rebuilt when deleted (for example if it has gotten corrupted)