1. Quantification of Integrity
Michael Clarkson and Fred B. Schneider
Cornell University
SnT Seminar
University of Luxembourg
September 9, 2010
Joint work with FBS at Cornell University.
The title says it all. This work is about quantification of integrity. But in more detail, I mean…

2. Goal
Information-theoretic Quantification of programs’ impact on Integrity of Information
[Denning 1982]
Idea of applying info theory to programs isn’t new. But looking at integrity is a quite recent development.
And what *is* integrity?

3. What is Integrity? Common Criteria: Biba (1977): Databases:
Protection of assets from unauthorized modification
Biba (1977):
Guarantee that a subsystem will perform as it was intended
Isolation necessary for protection from subversion
Dual to confidentiality
Databases:
Constraints that relations must satisfy
Provenance of data
Utility of anonymized data
…no universal definition
//Biba said “adheres to well-defined code of conduct” by analogy to personal integrity.
So we take 2 starting points.

4. Our Notions of Integrity
Corruption: damage to integrity
Starting Point
Corruption Measure
Taint analysis
Contamination
Program correctness
Suppression

5. Our Notions of Integrity
Corruption: damage to integrity
Starting Point
Corruption Measure
Taint analysis
Contamination
Program correctness
Suppression
…I’ll start by talking about contamination.
Contamination: bad information present in output
Suppression: good information lost from output
…distinct, but interact

6. Contamination Goal: model taint analysis Program untrusted Attacker
User
User
Untrusted = tainted; trusted = untainted.
This model has active attacker who provides inputs, not just an attacker who observes output.

7. Untrusted input contaminates trusted output
Contamination
Goal: model taint analysis
Untrusted input contaminates trusted output
Program
untrusted
Attacker
Attacker
trusted
User
User
If this looks familiar---it should.

8. Contamination
u contaminates o
o:=(t,u)

9. (Can’t u be filtered from o?)
Contamination
u contaminates o
(Can’t u be filtered from o?)
o:=(t,u)
Answer: Yes. Just like in real life, contaminants can be filtered out. But contamination is about the amount of contaminant present in the output---not about what user does with output.

10. Quantification of Contamination
Use information theory: information is surprise X, Y, Z: distributions I(X,Y): mutual information between X and Y (in bits) I(X,Y | Z): conditional mutual information
An event with probability ½ conveys 1 bit of information. E.g., coin flip.

11. Quantification of Contamination
Program
User
Attacker
trusted
untrusted

12. Quantification of Contamination
Uin
Program
User
Attacker
trusted
untrusted
Tin
Tout
Uin and Tin characterize probability with which agents choose inputs.
Tout is a function of the input distributions and the program, which might itself behave randomly.

13. Quantification of Contamination
Contamination = I(Uin,Tout | Tin)
Uin
Program
User
Attacker
trusted
untrusted
Tin
Tout
Why condition on Tin? To model the fact that the user can observe the trusted inputs---since he does supply them himself.
[Newsome et al. 2009]
Dual of [Clark et al. 2005, 2007]

14. Example of Contamination
o:=(t,u)
Contamination = I(U, O | T) = k bits
if U is uniform on [0,2k-1]

15. Our Notions of Integrity
Corruption: damage to integrity
Starting Point
Corruption Measure
Taint analysis
Contamination
Program correctness
Suppression
Contamination: bad information present in output
Suppression: good information lost from output

16. Program Suppression Goal: model program (in)correctness Specification
Sender
Receiver
correct
(Specification must be deterministic)

17. Program Suppression Goal: model program (in)correctness Specification
Sender
Receiver
correct
Implementation
untrusted
Attacker
Attacker
Attacker and Sender could share choosing the Spec’s Inputs.
How to divide Input between Attacker and Sender? Doesn’t matter; just need joint probability distribution on Spec and Impl to ensure that inputs are the same in individual runs.
Likewise, can match up differing named, typed outputs via joint distribution.
trusted
Sender
Receiver
real

18. Program Suppression Goal: model program (in)correctness
Specification
Sender
Receiver
correct
Implementation
untrusted
Attacker
Attacker
Implementation might be buggy; attacker might influence output.
Normally think of any incorrectness as bad. Here, we quantify---more nuanced.
trusted
Sender
Receiver
real
Implementation might suppress information about correct output from real output

19. Example of Program Suppression
Spec.
for (i=0; i<m; i++)
{ s := s + a[i]; }
a[0..m-1]: trusted

20. Example of Program Suppression
Spec.
for (i=0; i<m; i++)
{ s := s + a[i]; }
a[0..m-1]: trusted
Impl. 1
for (i=1; i<m; i++)
{ s := s + a[i]; }
Suppression—a[0] missing
No contamination

21. Example of Program Suppression
Spec.
for (i=0; i<m; i++)
{ s := s + a[i]; }
a[0..m-1]: trusted
Impl. 1
Impl. 2
for (i=1; i<m; i++)
{ s := s + a[i]; }
for (i=0; i<=m; i++)
{ s := s + a[i]; }
a[m]: untrusted
Suppression—a[0] missing
No contamination
Suppression—a[m] added
Contamination

22. Suppression vs. Contamination
output := input
Contamination
Attacker
Attacker * *
Suppression

23. Quantification of Program Suppression
Specification
Sender
Receiver
Implementation
untrusted
Attacker
Attacker
trusted
Sender
Receiver

24. Quantification of Program SuppressionIn
Spec
Specification
Sender
Receiver
Implementation
untrusted
Attacker
Attacker
trusted
Sender
Receiver

25. Quantification of Program SuppressionIn
Spec
Specification
Sender
Receiver
Uin
Implementation
untrusted
Attacker
Attacker
trusted
Sender
Receiver
Tin
Impl

26. Quantification of Program SuppressionIn
Spec
Specification
Sender
Receiver
Uin
Implementation
untrusted
Attacker
Attacker
trusted
Sender
Receiver
Tin
Impl
Program transmission = I(Spec , Impl)

27. Quantification of Program Suppression
H(X): entropy (uncertainty) of X
H(X|Y): conditional entropy of X given Y
Program Transmission = I(Spec, Impl)
= H(Spec) − H(Spec | Impl)

28. Quantification of Program Suppression
H(X): entropy (uncertainty) of X
H(X|Y): conditional entropy of X given Y
Program Transmission = I(Spec, Impl)
= H(Spec) − H(Spec | Impl)
Total info to learn about Spec

29. Quantification of Program Suppression
H(X): entropy (uncertainty) of X
H(X|Y): conditional entropy of X given Y
Program Transmission = I(Spec, Impl)
= H(Spec) − H(Spec | Impl)
Info actually learned about Spec by observing Impl
Total info to learn about Spec

30. Quantification of Program Suppression
H(X): entropy (uncertainty) of X
H(X|Y): conditional entropy of X given Y
Program Transmission = I(Spec, Impl)
= H(Spec) − H(Spec | Impl)
Info actually learned about Spec by observing Impl
Total info to learn about Spec
Info NOT learned about Spec by observing Impl

31. Quantification of Program Suppression
H(X): entropy (uncertainty) of X
H(X|Y): conditional entropy of X given Y
Program Transmission = I(Spec, Impl)
= H(Spec) − H(Spec | Impl)
Program Suppression = H(Spec | Impl)

32. Example of Program Suppression
Spec.
for (i=0; i<m; i++)
{ s := s + a[i]; }
Impl. 1
Impl. 2
for (i=1; i<m; i++)
{ s := s + a[i]; }
for (i=0; i<=m; i++)
{ s := s + a[i]; }
Of course, we can quantify suppression of impl. 2 exactly, given specific probability distributions.
I’ve defined contamination and suppression now for mutual information based definitions. Can extend to richer scenario of beliefs.
Suppression = H(A)
Suppression ≤ H(A)
A = distribution of individual array elements

33. Echo Specification
Tin
Tin
output := input
trusted
Sender
Receiver

34. Echo Specification Tin Tin output := input trusted Sender Receiver Uin
Implementation
untrusted
Attacker
Attacker
trusted
Sender
Receiver
Tin
Tout

35. Simplifies to information-theoretic model of channels, with attacker
Echo Specification
Tin
Tin
output := input
trusted
Sender
Receiver
Uin
Implementation
untrusted
Attacker
Attacker
trusted
Sender
Receiver
Tin
Tout
Simplifies to information-theoretic model of channels, with attacker

36. Channel Suppression Channel transmission = I(Tin,Tout)
Uin
Channel
untrusted
Attacker
Attacker
trusted
Sender
Receiver
Tin
Tout
Channel transmission = I(Tin,Tout)
Channel suppression = H(Tin | Tout)
(Tout depends on Uin)

37. Belief-based Metrics
What if user’s/receiver’s distribution on unobservable inputs is wrong?
 Belief-based information flow [Clarkson et al. 2005]
Belief-based generalizes information-theoretic:
On single executions, the same
In expectation, the same …if user’s/receiver’s distribution is correct
Correct means that the distribution is NOT wrong.
What about relationship between confidentiality and integrity?

38. Suppression and Confidentiality
Declassifier: program that reveals (leaks) some information; suppresses rest
Leakage: [Denning 1982, Millen 1987, Gray 1991, Lowe 2002, Clark et al. 2005, 2007, Clarkson et al. 2005, McCamant & Ernst 2008, Backes et al. 2009]
Thm. Leakage + Suppression is a constant
 What isn’t leaked is suppressed
This is for a single execution. Constant determined by distribution on secret inputs---in particular, I(h) where h is the high input event for the execution.

39. Database Privacy
Statistical database anonymizes query results: …sacrifices utility for privacy’s sake
response
Anonymizer
Database
User
User
query
anonymized response

40. Database Privacy
Statistical database anonymizes query results: …sacrifices utility for privacy’s sake …suppresses to avoid leakage
response
Anonymizer
Database
User
User
query
anonymized response
anon. resp. := resp.

41. Database Privacy
Statistical database anonymizes query results: …sacrifices utility for privacy’s sake …suppresses to avoid leakage …sacrifices integrity for confidentiality’s sake
response
Anonymizer
Database
User
User
query
anonymized response

42. K-anonymity
DB. Every individual must be anonymous within set of size k [Sweeney 2002]
Iflow. Every output corresponds to k inputs
…no bound on leakage or suppression

43. L-diversity
DB. Every individual’s sensitive information should appear to have L (roughly) equally likely values [Machanavajjhala et al. 2007] DB. Entropy L-diversity: H(anon. block) ≥ log L [Øhrn and Ohno-Machado 1999, Machanavajjhala et al. 2007] Iflow. H(Tin | tout) ≥ log L (if Tin uniform) …implies suppression ≥ log L

44. Differential Privacy DB. [Dwork et al. 2006, Dwork 2006]
No individual loses privacy by including data in database
Anonymized data set reveals no information about an individual beyond what other individuals already reveal
Iflow. Output reveals almost no information about individual input beyond what other inputs already reveal
…implies almost all information about individual suppressed
…quite similar to noninterference

45. Summary Measures of information corruption:
Contamination (generalizes taint analysis, dual to leakage)
Suppression (generalizes program correctness, no dual)
Application: database privacy
(model anonymizers; relate utility and privacy; security conditions)

46. More Integrity Measures
Attacker- and program-controlled suppression
Granularity:
Average over all executions
Single executions
Sequences of executions
…interaction of attacker with program
Application: Error-correcting codes

47. Quantification of Integrity
Michael Clarkson and Fred B. Schneider
Cornell University
SnT Seminar
University of Luxembourg
September 9, 2010
Joint work with FBS at Cornell University.
The title says it all. This work is about quantification of integrity. But in more detail, I mean…