Anupam Datta Anupam DattaCMU Joint work with Adam Barth, John Mitchell (Stanford), Helen Nissenbaum (NYU) and Sharada Sundaram (TCS) Privacy and Contextual Integrity: Framework and Applications

2 Problem Statement Is an organization’s business process compliant with privacy regulations and internal policies? Examples of organizations – Hospitals, financial institutions, other enterprises handling sensitive information Examples of privacy regulations – HIPAA, GLBA, COPPA, SB1386 Goal: Develop methods and tools to answer this question

3 Privacy Project Space What is Privacy? [Philosophy, Law, Public Policy] Formal Model, Policy Language, Compliance-check Algorithms [Programming Languages, Logic] Implementation-level Compliance [Software Engg, Formal Methods] Data Privacy [Databases, Cryptography]

4 Project Overview What is privacy? – Conceptual framework Policy language – Privacy laws including HIPAA, COPPA, GLBA expressible Compliance-check algorithms – Does system satisfy privacy and utility goals? Case studies – Patient portal deployed at Vanderbilt Hospital – UPMC (ongoing discussions) – TCS

5 Contextual Integrity [N2004] Philosophical framework for privacy Central concept: Context – Examples: Healthcare, banking, education What is a context? – Set of interacting agents in roles Roles in healthcare: doctor, patient, … – Norms of transmission Doctors should share patient health information as per the HIPAA rules – Purpose Improve health

6 Nurse Secretary Workflow Patient Doctor Health Answer Health Question Appointment Request Health Question Now that I have cancer, Should I eat more vegetables? Yes! except broccoli Privacy: HIPAA compliance+ Humans + Electronic system Utility: Schedule appointments, obtain health answers

7 Nurse Secretary Improved Patient Doctor Health Answer Health Question Appointment Request Health Question Now that I have cancer, Should I eat more vegetables? Health Question Yes! except broccoli Health Answer Message tags used for policy enforcement Minimal disclosure Responsibility: Doctor should answer health questions

8 Privacy vs. Utility Privacy – Certain information should not be communicated Utility – Certain information should be communicated Tension between privacy and utility Permissiveness Workflows Violate Privacy Violate Utility Feasible Workflows “Minimum necessary”

9 Design-time Analysis: Big Picture Contextual Integrity Business ObjectivesPrivacy Policy Business Process Design Privacy Checker (LTL) Utility Checker (ATL*) Utility Evaluation Privacy Evaluation Norms Purpose Assuming agents responsible

10 Auditing: Big Picture Business Process Execution Audit Logs Run-time Monitor Privacy Policies Utility Goals Audit Algos Policy Violation + Accountable Agent Agents may not be responsible

11 In more detail… Model and logic Privacy policy examples – GLBA – financial institutions – MyHealth portal Compliance checking – Design time analysis (fully automated) – Auditing (using oracle) Language can express HIPAA, GLBA, COPPA [BDMN2006]

12 Model Alice – Communication via send actions: Sender: Bob in role Patient Recipient: Alice in role Nurse Subject of message: Bob Tag: Health Question Message: Now that …. – Data model & knowledge evolution: Agents acquire knowledge by: – receiving messages – deriving additional attributes based on data model Health Question  Protected Health Information Bob Now that I have cancer, Should I eat more vegetables? Health Question contents(msg) vs. tags (msg) Inspired by Contextual Integrity

13 Model State determined by knowledge of each agent Transitions change state – Set of concurrent send actions – Send(p,q,m) possible only if agent p knows m K0K0 K 13 K K 12 A 11 A 12 A 13 Concurrent Game Structure G = [BDMN06, BDMS07]

14 Logic of Privacy and Utility Syntax  ::= send(p 1,p 2,m)p 1 sends p 2 message m | contains(m, q, t)m contains attrib t about q | tagged(m, q, t)m tagged attrib t about q | inrole(p, r)p is active in role r | t  t’Attrib t is part of attrib t’ |    |  |  x.  Classical operators |  U  |  S  | O  Temporal operators | >  Strategy quantifier Semantics Formulas interpreted over concurrent game structure

15 Specifying Privacy In all states, only nurses and doctors receive health questions G  p1, p2, q, m send(p1, p2, m)  contains(m, q, health-question)  inrole(p2, nurse)  inrole(p2, doctor)

16 Specifying Utility Patients have a strategy to get their health questions answered  p inrole(p, patient)  > F  q, m. send(q, p, m)  contains(m, p, health-answer)

17 MyHealth Responsibilities Tagging Nurses should tag health questions G  p, q, s, m. inrole(p, nurse)  send(p, q, m)  contains(m, s, health-question)  tagged(m, s, health-question) Progress – Doctors should answer health questions G  p, q, s, m. inrole(p, doctor)  send(q, p, m)  contains(m, s, health-question)  F  m’. send(p, s, m’)  contains(m’, s, health-answer)

18 Sender roleSubject roleAttribute Temporal condition Gramm-Leach-Bliley Example Recipient role Financial institutions must notify consumers if they share their non-public personal information with non- affiliated companies, but the notification may occur either before or after the information sharing occurs

19 Workflow Design Results Theorems: Assuming all agents act responsibly, checking whether workflow achieves – Privacy is in PSPACE (in the size of the formula describing the workflow) Use LTL model-checking algorithm – Utility is decidable for a restricted class of formulas ATL* model-checking is undecidable for concurrent game structures with imperfect information, but decidable with perfect information Idea: – Check that all executions satisfy privacy and utility properties Definition and construction of minimal disclosure workflow Algorithms implemented in model-checkers, e.g. SPIN, MOCHA

20 Auditing Results Who to blame? Accountability – Irresponsibility + causality Design of audit log – Use Lamport causality structure, standard concept from distributed computing Algorithms – Finding agents accountable for policy violation in graph-based workflows using audit log – Finding agents who act irresponsibly using audit log Algorithms use oracle: – O(msg) = contents(msg) – Minimize number of oracle calls

21 Conclusions Framework inspired by contextual integrity Business Process as Workflow u Role-based responsibility for human and mechanical agents Compliance checking u Workflow design assuming agents responsible u Privacy, utility decidable (model-checking) u Minimal disclosure workflow constructible u Auditing logs when agents irresponsible u From policy violation to accountable agents u Finding irresponsible agents Case studies – MyHealth patient portal deployed at Vanderbilt University hospital – Ongoing interactions with UPMC Using oracle Automated

22 Future Work Framework – Do we have the right concepts? – Adding time, finer-grained data model – Priorities of rules, inconsistency, paraconsistency Compliance vs. risk management Privacy principles – Minimum necessary one example; what else? Improve algorithmic results – Utility decidability; small model theorem – Auditing algorithms Privacy analysis of code – Current results apply to system specification More case studies – Focusing on healthcare Detailed specification of privacy laws – Immediate focus on HIPAA, GLBA, COPPA Legal, economic incentives for responsible behavior

23 Publications/Credits – A. Barth, A. Datta, J. C. Mitchell, S. Sundaram Privacy and Utility in Business Processes, to appear in Proceedings of 20th IEEE Computer Security Foundations Symposium, July – A. Barth, A. Datta, J. C. Mitchell, H. Nissenbaum Privacy and Contextual Integrity: Framework and Applications, in Proceedings of 27th IEEE Symposium on Security and Privacy, pp , May Work covered in The Economist, IEEE Security & Privacy editorial

24 Thanks Questions?

25 Additional Technical Slides

26 Related Languages ModelSenderRecipientSubjectAttributesPastFutureCombination RBACRoleIdentity  XACMLFlexible o  o  EPALFixedRoleFixed  o  P3PFixedRoleFixed  o  o LPURole  Legend:  unsupported opartially supported  fully supported LPU fully supports attributes, combination, temporal conditions Utility not considered

27 Deciding Utility ATL* model-checking of concurrent game structures is – Decidable with perfect information – Undecidable with imperfect information Theorem: There is a sound decision procedure for deciding whether workflow achieves utility Intuition: – Translate imperfect information into perfect information by considering all possible actions from one player’s point of view

28 Local communication game Quotient structure under invisible actions, G p – States: Smallest equivalence relation K 1 ~ p K 2 if K 1  K 2 and a is invisible to p – Actions: [K]  [K’] if there exists K 1 in [K] and K 2 in [K’] s.t. K 1  K 2 Lemma: For all LTL formulas  visible to p, G p |= >  implies G |= > 

29 Auditing Results Definitions – Policy compliance, locally compliant – Causality, accountability Design of audit log Algorithms – Finding agents accountable for locally-compliant policy violation in graph-based workflows using audit log – Finding agents who act irresponsibly using audit log Algorithms use oracle: – O(msg) = contents(msg) – Minimize number of oracle calls

30 Policy compliance/violation Strong compliance [BDMN2006] – Action does not violate current policy requirements – Future policy requirements after action can be met Locally compliant policy – Agents can determine strong compliance based on their local view of history Policy History Contemplated Action Judgment Future Reqs

31 Causality Lamport Causality [1978] “happened-before”

32 Accountability & Audit Log Accountability – Causality + Irresponsibility Audit log design – Records all Send(p,q,m) and Receive(p,q,m) events executed – Maintains causality structure O(1) operation per event logged

33 Auditing Algorithm Goal Find agents accountable for a policy violation Algorithm(Audit log A, Violation v) 1. Construct G, the causality graph for v in A 2. Run BFS on G. At each Send(p, q, m) node, check if tags(m) = O(m). If not, and p missed a tag, output p as accountable Theorem: – The algorithm outputs at least one accountable agent for every violation of a locally compliant policy in an audit log of a graph-based workflow that achieves the policy in the responsible model

34 Proof Idea Causality graph G includes all accountable agents – Accountability = Causality + Irresponsibility There is at least one irresponsible agent in G – Policy is satisfied if all agents responsible – Policy is locally compliant In graph-based workflows, safety responsibilities violated only by mistagging – O(msg) = tags(msg) check identifies all irresponsible actions

35 MyHealth Example 1. Policy violation: Secretary Candy receives health-question mistagged as appointment-request 2. Construct causality graph G and search backwards using BFS Candy received message m from Patient Jorge. u O(m) = health-question, but tags(m) = appointment-request. u Patient responsible for health-question tag. u Jorge identified as accountable