Anupam Datta

1
Privacy and Contextual IntegrityFramework and
Applications

• Anupam Datta
• CMU
• Joint work with Adam Barth, John Mitchell
  (Stanford), Helen Nissenbaum (NYU) and Sharada
  Sundaram (TCS)

2
Problem Statement

• Is an organizations business process compliant
  with privacy regulations and internal policies?
• Examples of organizations
• Hospitals, financial institutions, other
  enterprises handling sensitive information
• Examples of privacy regulations
• HIPAA, GLBA, COPPA, SB1386

Goal Develop methods and tools to answer this
question
3
Privacy Project Space
What is Privacy? Philosophy, Law, Public Policy
Formal Model, Policy Language, Compliance-check
Algorithms Programming Languages, Logic
Implementation-level Compliance Software Engg,
Formal Methods
Data Privacy Databases, Cryptography
4
Project Overview

• What is privacy?
• Conceptual framework
• Policy language
• Privacy laws including HIPAA, COPPA, GLBA
  expressible
• Compliance-check algorithms
• Does system satisfy privacy and utility goals?
• Case studies
• Patient portal deployed at Vanderbilt Hospital
• UPMC (ongoing discussions)
• TCS

5
Contextual Integrity N2004

• Philosophical framework for privacy
• Central concept Context
• Examples Healthcare, banking, education
• What is a context?
• Set of interacting agents in roles
• Roles in healthcare doctor, patient,
• Norms of transmission
• Doctors should share patient health information
  as per the HIPAA rules
• Purpose
• Improve health

6
MyHealth_at_Vanderbilt Workflow
Health Answer
Humans Electronic system
Yes! except broccoli
Appointment Request
Secretary
Health Question
Health Question
Now that I have cancer, Should I eat more
vegetables?
Doctor
Patient
Health Question
Health Answer
Utility Schedule appointments, obtain health
answers
Nurse
Privacy HIPAA compliance
7
MyHealth_at_Vanderbilt Improved
Health Answer
Appointment Request
Secretary
Doctor
Patient
Health Question
Health Question

• Message tags used for policy enforcement
• Minimal disclosure

Health Answer
Nurse
Responsibility Doctor should answer health
questions
8
Privacy vs. Utility

• Privacy
• Certain information should not be communicated
• Utility
• Certain information should be communicated
• Tension between privacy and utility

Workflows
Violate Privacy
Feasible Workflows
Permissiveness
Minimum necessary
Violate Utility
9
Design-time Analysis Big Picture
Norms
Purpose
Business Objectives
Privacy Policy
Privacy Checker (LTL)
Utility Checker (ATL)
Business Process Design
Utility Evaluation
Privacy Evaluation
Assuming agents responsible
10
Auditing Big Picture
Business Process Execution
Run-time Monitor
Audit Logs
Audit Algos
Privacy Policies Utility Goals
Policy Violation Accountable Agent
Agents may not be responsible
11
In more detail

• Model and logic
• Privacy policy examples
• GLBA financial institutions
• MyHealth portal
• Compliance checking
• Design time analysis (fully automated)
• Auditing (using oracle)

Language can express HIPAA, GLBA, COPPA
BDMN2006
12
Model
Inspired by Contextual Integrity
Alice
Bob

• Communication via send actions
• Sender Bob in role Patient
• Recipient Alice in role Nurse
• Subject of message Bob
• Tag Health Question
• Message Now that .
• Data model knowledge evolution
• Agents acquire knowledge by
• receiving messages
• deriving additional attributes based on data
  model
• Health Question ? Protected Health Information

contents(msg) vs. tags (msg)
13
Model
BDMN06, BDMS07

• State determined by knowledge of each agent
• Transitions change state
• Set of concurrent send actions
• Send(p,q,m) possible only if agent p knows m

K0
A13
A11
K13
A12
K11
...
K12
...
Concurrent Game Structure G ltk, Q, ?, ?, d, ?gt
14
Logic of Privacy and Utility

• Syntax
• ? send(p1,p2,m) p1 sends p2 message m
• contains(m, q, t) m contains attrib t
  about q
• tagged(m, q, t) m tagged attrib t about
  q
• inrole(p, r) p is active in role r
• t ? t Attrib t is part of attrib t
• ? ? ? ?? ?x. ? Classical operators
• ?U? ?S? O? Temporal operators
• ltltpgtgt? Strategy quantifier
• Semantics
• Formulas interpreted over concurrent game
  structure

15
Specifying Privacy

• MyHealth_at_Vanderbilt
• In all states, only nurses and doctors
  receive health questions
• G ? p1, p2, q, m
• send(p1, p2, m) ? contains(m, q, health-question)
• ? inrole(p2, nurse) ? inrole(p2, doctor)

16
Specifying Utility

• MyHealth_at_Vanderbilt
• Patients have a strategy to get their health
  questions answered
• ? p inrole(p, patient) ?
• ltltpgtgt F ? q, m.
• send(q, p, m) ? contains(m, p,
  health-answer)

17
MyHealth Responsibilities

• Tagging
• Nurses should tag health questions
• G ?p, q, s, m. inrole(p, nurse) ? send(p, q, m) ?
  contains(m, s, health-question)
• ? tagged(m, s, health-question)
• Progress
• Doctors should answer health questions
• G ?p, q, s, m. inrole(p, doctor) ? send(q, p, m)
  ? contains(m, s, health-question) ?
• F ?m. send(p, s, m) ?
• contains(m, s, health-answer)

18
Gramm-Leach-Bliley Example
Financial institutions must notify consumers if
they share their non-public personal information
with non-affiliated companies, but the
notification may occur either before or after the
information sharing occurs
19
Workflow Design Results

• Theorems
• Assuming all agents act responsibly, checking
  whether workflow achieves
• Privacy is in PSPACE (in the size of the formula
  describing the workflow)
• Use LTL model-checking algorithm
• Utility is decidable for a restricted class of
  formulas
• ATL model-checking is undecidable for concurrent
  game structures with imperfect information, but
  decidable with perfect information
• Idea
• Check that all executions satisfy privacy and
  utility properties
• Definition and construction of minimal disclosure
  workflow

Algorithms implemented in model-checkers, e.g.
SPIN, MOCHA
20
Auditing Results

• Who to blame? Accountability
• Irresponsibility causality
• Design of audit log
• Use Lamport causality structure, standard concept
  from distributed computing
• Algorithms
• Finding agents accountable for policy violation
  in graph-based workflows using audit log
• Finding agents who act irresponsibly using audit
  log
• Algorithms use oracle
• O(msg) contents(msg)
• Minimize number of oracle calls

21
Conclusions

• Framework inspired by contextual integrity
• Business Process as Workflow
• Role-based responsibility for human and
  mechanical agents
• Compliance checking
• Workflow design assuming agents responsible
• Privacy, utility decidable (model-checking)
• Minimal disclosure workflow constructible
• Auditing logs when agents irresponsible
• From policy violation to accountable agents
• Finding irresponsible agents
• Case studies
• MyHealth patient portal deployed at Vanderbilt
  University hospital
• Ongoing interactions with UPMC

Automated
Using oracle
22
Future Work

• Framework
• Do we have the right concepts?
• Adding time , finer-grained data model
• Priorities of rules, inconsistency,
  paraconsistency
• Compliance vs. risk management
• Privacy principles
• Minimum necessary one example what else?
• Improve algorithmic results
• Utility decidability small model theorem
• Auditing algorithms
• Privacy analysis of code
• Current results apply to system specification
• More case studies
• Focusing on healthcare
• Detailed specification of privacy laws
• Immediate focus on HIPAA, GLBA, COPPA
• Legal, economic incentives for responsible
  behavior

23
Publications/Credits

• A. Barth, A. Datta, J. C. Mitchell, S. Sundaram
• Privacy and Utility in Business Processes, to
  appear in Proceedings of 20th IEEE Computer
  Security Foundations Symposium, July 2007.
• A. Barth, A. Datta, J. C. Mitchell, H. Nissenbaum
  
• Privacy and Contextual Integrity Framework
  and Applications, in Proceedings of 27th IEEE
  Symposium on Security and Privacy , pp. 184-198,
  May 2006.

Work covered in The Economist, IEEE Security
Privacy editorial
24
Thanks Questions?
25
Additional Technical Slides
26
Related Languages

• Legend
• ? unsupported
• o partially supported
• ? fully supported
• LPU fully supports attributes, combination,
  temporal conditions

Utility not considered
27
Deciding Utility

• ATL model-checking of concurrent game structures
  is
• Decidable with perfect information
• Undecidable with imperfect information
• Theorem
• There is a sound decision procedure for deciding
  whether workflow achieves utility
• Intuition
• Translate imperfect information into perfect
  information by considering all possible actions
  from one players point of view

28
Local communication game

• Quotient structure under invisible actions, Gp
• States
• Smallest equivalence relation
• K1 p K2 if K1 ? K2 and a is invisible to p
• Actions
• K ? K if there exists K1 in K and K2 in
  K s.t. K1 ? K2
• Lemma For all LTL formulas ?? visible to p, Gp
  ltltpgtgt? implies G ltltpgtgt?

29
Auditing Results

• Definitions
• Policy compliance, locally compliant
• Causality, accountability
• Design of audit log
• Algorithms
• Finding agents accountable for locally-compliant
  policy violation in graph-based workflows using
  audit log
• Finding agents who act irresponsibly using audit
  log
• Algorithms use oracle
• O(msg) contents(msg)
• Minimize number of oracle calls

30
Policy compliance/violation
Contemplated Action
Judgment
Policy
Future Reqs
History

• Strong compliance
  BDMN2006
• Action does not violate current policy
  requirements
• Future policy requirements after action can be
  met
• Locally compliant policy
• Agents can determine strong compliance based on
  their local view of history

31
Causality

• Lamport Causality
• 1978
• happened-before

32
Accountability Audit Log

• Accountability
• Causality Irresponsibility
• Audit log design
• Records all Send(p,q,m) and Receive(p,q,m) events
  executed
• Maintains causality structure
• O(1) operation per event logged

33
Auditing Algorithm

• Goal
• Find agents accountable for a policy violation
• Algorithm(Audit log A, Violation v)
• Construct G, the causality graph for v in A
• Run BFS on G.
• At each Send(p, q, m) node, check if tags(m)
  O(m). If not, and p missed a tag, output p as
  accountable
• Theorem
• The algorithm outputs at least one accountable
  agent for every violation
• of a locally compliant policy in an audit log
• of a graph-based workflow that achieves the
  policy in the responsible model

34
Proof Idea

• Causality graph G includes all accountable agents
• Accountability Causality Irresponsibility
• There is at least one irresponsible agent in G
• Policy is satisfied if all agents responsible
• Policy is locally compliant
• In graph-based workflows, safety
  responsibilities violated only by mistagging
• O(msg) tags(msg) check identifies all
  irresponsible actions

35
MyHealth Example

• Policy violation
• Secretary Candy receives health-question
  mistagged as appointment-request
• Construct causality graph G and search backwards
  using BFS
• Candy received message m from Patient Jorge.
  
• O(m) health-question, but tags(m)
  appointment-request.
• Patient responsible for health-question tag.
• Jorge identified as accountable