1. Hyperledger Fabric Private Data - Collection Types

2. Types of collections
Collection type
Distribution
Use Case
This is just normal chaincode data. Technically, there is no such thing as a ‘public collection’, it’s just public chaincode data, but including it here for comparison.
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
A collection is a set of private data in a chaincode namespace.
A collection is defined by it’s data distribution policy (that is, collection membership, in all cases except for local collections), and has it’s own namespace within the chaincode.
Key structure in state db: channel~chaincode~collection~key
The different types of collections are based on different types of distribution (only).
Local
Client
In v1.2 plan
Who distributes
private data to orgs?
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
All
Implicit definition
Idea to consider
Private
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
Immutable
Membership
Peers
Idea to consider
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
Mutable
Membership*
Explicit def
In v1.2 plan
*Note, another variation of Mutable Membership could create a collection dynamically (new API required) and distribute members as private data (in a local collection), and then going forward peers of these organizations use the private collection definition to distribute private data instead of the client doing it.
Idea to consider

3. Example - three orgs, plus a regulator
In this example, a regulator gets all private data for audit purpose.
Collection type
Distribution
Use Case
OrgA
OrgB
OrgC
Regulator
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
Private data distribution per transaction.
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
Local
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
All
Implicit definition
Private
private data –
has access
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
AB R
AB R
AB R
AB R
Immutable
Membership
BC R
BC R
BC R
BC R
private data –
no access,
but has public hashes for validation
AC R
AC R
AC R
AC R
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
AB R
AB R
AB R
AB R
Mutable
Membership
Explicit def
BC R
BC R
BC R
BC R
AC R
AC R
AC R
AC R
Private data distribution per collection.
OrgC gets all AC and BC data.

4. Collection definition – implicit vs explicit
Collection type
Distribution
Use Case
OrgA
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
There is one implicit ‘local’ collection per chaincode.
In chaincode, specify empty string for collection name.
Local
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
There is one implicit ‘all’ collection per chaincode.
In chaincode, specify ‘*’ for collection name.
All
Implicit definition
Note – This is only an idea for consideration at this point.
Private
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
AB R
There is an implicit collection for each unique combination of organizations in the channel.
Client specifies list of MSPIDs for distribution.
For the ledger collection namespace, use a hash of the MSPIDs.
Immutable
Membership
BC R
Note – This is only an idea for consideration at this point.
AC R
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
AB R
Mutable
Membership
Explicit def
Explicitly specify named collections with specific MSPID memberships at chaincode instantiation time (aka definition). Memberships can change with each chaincode version.
In chaincode, specify collection name.
BC R
AC R
Only the last type of collection has an explicit named definition which needs to be created.

5. Collection definition – org local state
Collection type
Distribution
Use Case
OrgA
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
Note the special case:
If a single MSPID is specified, this indicates local state for this org only.
Useful for ‘org local state’ such as chaincode metadata package, org-specific approvals, thresholds, voting, etc.
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
Local
These are implict combinations of any one org,
aka,
‘org local state.’ A
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
All B
Implicit definition C
Private
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
AB R
There is an implicit collection for each unique combination of organizations in the channel.
Client specifies list of MSPIDs for distribution.
For the ledger collection namespace, use a hash of the MSPIDs.
Immutable
Membership
BC R
AC R
Note – This is only an idea for consideration at this point. In the short term, we would likely only have implicit support for the single org collections.
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
AB R
Mutable
Membership
Explicit def
BC R
These are implicit combinations of any three orgs. These collections are implicit (does not need to be defined) and immutable (will never change).
AC R

6. Collection namespaces
Collection type
Distribution
Use Case
OrgA
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
The implicit ‘local’ collection uses a single namespace. All keys must be unique.
Facilitates asset transfer to any org in channel, while preserving provenance.
Local
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
The implicit ‘all’ collection uses a single namespace. All keys must be unique.
Facilitates asset transfer to any org in channel, while preserving provenance.
All
Implicit definition
Private
There is a namespace per implicit collection within the chaincode.
Keys must be unique within each collection. This implies that collection members ‘own’ the namespace and can create any key. e.g. for the single MSPID collections, this is a reserved namespace for the org to write their keys.
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
AB R
Immutable
Membership
BC R
AC R
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
AB R
Mutable
Membership
Explicit def
There is a namespace per explicit collection within the chaincode.
Keys must be unique within each collection. This implies that collection members ‘own’ the namespace and can create any key.
BC R
AC R
Example namespace:
channel~chaincode~collectionACR

7. State-based endorsement
Use consistent state-based endorsement approach across all collection types, to simplify code and usage, and to faciliate simple asset transfer.
If we are worried about the extra per-record storage space, this can be a reference to an endorsement policy.
Collection type
Distribution
Use Case
OrgA
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
Endorsement policy can be stored in the key’s value as metadata.
Facilitates asset transfer to any org in channel, while preserving provenance.
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
Endorsement policy can be stored in the key’s value as metadata.
Facilitates asset transfer to any org in channel, while preserving provenance.
Local
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
Endorsement policy can be stored in the key’s value as metadata.
Facilitates asset transfer to any org in channel, with provenance.
All
Implicit definition
Private
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
AB R
Immutable
Membership
Endorsement policy can stored in the key’s value as metadata.
Facilitates asset transfer to orgs within same collection, while preserving provenance.
BC R
AC R
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
AB R
Endorsement policy can be stored in the key’s value as metadata.
Facilitates asset transfer to orgs within same collection, while preserving provenance.
Mutable
Membership
Explicit def
BC R
AC R
In the case of private data, the endorsement policy would actually be stored in the public statedb within the value of the hash(key), so that non-members can also perform validation.
= keys that OrgA ‘owns’ via state-based endorsement

8. Collection – query implications
read-only queries
read-write txs
Range query
JSON query
(CouchDB) ✓
no phantom validation for couch queries
allowed on own org’s peer only?
phantom validation not possible
within collections
within
and across
collections
Collection type
Distribution
Use Case
OrgA
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
Local
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
All
Implicit definition
Private
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
AB R
Immutable
Membership
BC R
AC R
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
AB R
Mutable
Membership
Explicit def
BC R
AC R
One CouchDB database with 3 collection namespaces.
Can query one collection or entire database.

9. Collection – query implications
read-only queries
read-write txs
Range query
JSON query
(CouchDB) ✓
no phantom validation for couch queries
phantom validation not possible
within collections
within
and across
collections
Range queries re-executed at validation time to ensure no phantoms.
Collection type
Distribution
Use Case
OrgA
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
No phantom validation for couchdb queries, some applications can tolerate phantoms, others can’t
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
Local
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
All
Implicit definition
Range queries re-executed at validation time to ensure no phantoms. Note: only valid if we don’t allow missing data.
Private
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
AB R
Immutable
Membership
BC R
AC R
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
AB R
Mutable
Membership
Explicit def
BC R
AC R
Due to collection namespace, can’t do key range queries across collections …unless we duplicate the state data with two key structures: collection~key and key~collection.
With private data, peers that don’t have the data can’t validate the results
Can be supported at data layer ‘for free’, requires additional shim API.

10. Collection – query implications
read-only queries
read-write txs
Range query
JSON query
(CouchDB) ✓
no phantom validation for couch queries
phantom validation not possible
within collections
within
and across
collections
Collection type
Distribution
Use Case
OrgA
Public
Public data
(via ordered
blocks)
Normal public data
Normal public data
Client distributes private data to endorsing orgs, gossip shares within these orgs
Collection membership, data, and owner privacy;
asset transfer to any org
Local
Peer gossip distributes private data to all organizations
Keep private data confidential from orderers
All
Implicit definition
Private
How to handle?
Need a chaincode config option: allowReadWriteTransactionsWithoutPhantomCheck
true indicates to allow them (current behavior for CouchDB JSON queries)
false indicates to fail endorsement if this is detected
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between fixed set of orgs.
AB R
Immutable
Membership
BC R
AC R
Peer gossip distributes private data to all organizations in collection.
Data privacy; Share private data between evolving groups of orgs.
AB R
Mutable
Membership
Explicit def
BC R
AC R

11. Deeper dive on state-based ownership, local collections, and asset transfer.

12. With state-based endorsement, there may be a different endorsement policy for each item in the collection.
k AC {
JSON }
k BC {
JSON }
k AB {
JSON }
‘Endorsement policy’ gets added to value as metadata to support validation, similar to ‘version’.
Let’s simplify this by using a table view of the data… c1 c2 c3 EP k1
--- AB k2 BC k3 AC
AB = Owned jointly by OrgA/OrgB
BC = Owned jointly by OrgB/OrgC
AC = Owned jointly by OrgA/OrgC
*Options exist for linkable as well as unlinkable ownership (e.g. ephemeral keys per state, Idemix endorsement credentials)
For k1 with ownable state AB, client must send endorsement requests to OrgA and OrgB

13. With local collections, data may be private to a subset of orgs, based on which orgs did the endorsement for a particular transaction c1 c2 c3 EP k1
--- AB k3 AC
OrgA
Client
For k1 with ownable state AB, client must send endorsement requests to OrgA and OrgB. c1 c2 c3 EP k1
--- AB k2 BC
OrgB c1 c2 c3 EP k2
--- BC k3 AC
OrgC

14. With local collections, data may be private to a subset of orgs, based on which orgs did the endorsement for a particular transaction c1 c2 c3 EP k1
--- AB k3 AC
OrgA
Client
from OrgA
Clients are authorized to do non-key queries (range queries, couchdb json queries) against their own org only, results will include all private data that the org has access to.
In this example, client from OrgA queries OrgA peer for all data that OrgA has access to.
Note: non-key queries cannot be used in read/write transactions of private data, since it is impossible for other orgs to validate transactions for which they do not have the data. But client can do non-key query to understand state, and then propose key-based transactions. c1 c2 c3 EP k1
--- AB k2 BC
OrgB c1 c2 c3 EP k2
--- BC k3 AC
OrgC

15. Recall that with Side DB each org has private data (for endorsement) and public data (hashes for validation), so the local collection will actually look like this…
private
public c1 c2 c3 EP k1
--- AB k3 AC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC
OrgA
Client
private
public c1 c2 c3 EP k1
--- AB k2 BC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC
For k1 with ownable state AB, client must send endorsement requests to OrgA and OrgB.
Transactions are validated on all peers, using the hashes, version numbers, and endorsement policies in the public state.
OrgB
private
public c1 c2 c3 EP k2
--- BC k3 AC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC
OrgC

16. For change of EP, old EP and new EP must be fulfilled
For change of EP, old EP and new EP must be fulfilled. (this is often equivalent to changing ownable state, e.g. asset transfer) AB
private
public c1 c2 c3 EP k1
--- AB k3 AC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC k1
OrgA AC
Client
private
public c1 c2 c3 EP k1
--- AB k2 BC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC
For k1 transfer AB to AC, client must send endorsement requests to OrgA and OrgB and OrgC
OrgB
But wait, we have a problem!
OrgC does not have the private data,
and therefore cannot endorse transactions that read/write k1… proceed to next slide…
private
public c1 c2 c3 EP k2
--- BC k3 AC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC
OrgC

17. For change of EP, old EP and new EP must be fulfilled
For change of EP, old EP and new EP must be fulfilled. (this is often equivalent to changing ownable state, e.g. asset transfer) AB
private
public c1 c2 c3 EP k1
--- AB k3 AC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC k1
OrgA AC
Client
private
public c1 c2 c3 EP k1
--- AB k2 BC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC
…not to worry…
client can pass k1 data to OrgC in a
new transient field.
As part of endorsement, peer will ensure hash of the passed k1 matches h(k1) in public state, and will use this value for any calls to GetState(k1), and then can successfully endorse the transfer.
Open question: OrgC will not have the prior history. Can OrgC now retrieve the prior history of k1? How does it prove that it is entitled to it?
OrgB
k1/v1
private
public c1 c2 c3 EP k2
--- BC k3 AC
H(k)
H(v) EP
H(k1)
H(v1) AB
H(k2)
H(v2) BC
H(k3)
H(v3) AC
OrgC

18. Asset transferred What did we accomplish? Transferred a private asset,AB
private
public c1 c2 c3 EP k1
--- AC k3
H(k)
H(v) EP
H(k1)
H(v1) AC
H(k2)
H(v2) BC
H(k3)
H(v3) k1
OrgA AC
private
public
What did we accomplish?
Transferred a private asset,
with private verifiability via the hashes (likely a regulator/auditor would be involved to verify hashes against pre-images),
while maintaining queryability of data that each org has access to c1 c2 c3 EP k1
--- AC k2 BC
H(k)
H(v) EP
H(k1)
H(v1) AC
H(k2)
H(v2) BC
H(k3)
H(v3)
OrgB will no longer participate in endorsement and will not get future updates.
OrgB
private
public c1 c2 c3 EP k1
--- AC k2 BC k3
H(k)
H(v) EP
H(k1)
H(v1) AC
H(k2)
H(v2) BC
H(k3)
H(v3)
OrgC is now an owner and must endorse future transactions.
OrgC

19. Asset transfer – FutureAB
These are implict combinations of any one org,
aka,
‘org local state.’ k1 A B AC C
Using the implicit collection for org-local state data, organizations will be able to set approval criteria.
Organizations can set org-specific state that can serve as pre-approval instructions, used in chaincodes, or as conditions for endorsement.
If proposal does not meet criteria, peer will not endorse.
Examples:
pre-approve asset transfer from AB to AC.
pre-approve any trade if sale price is greater than some limit