The Consequences of Decentralized Security in a Cooperative Storage System

1
The Consequences ofDecentralized Security in a
Cooperative Storage System

• Douglas Thain, Chris Moretti, Paul Madrid,
• Phil Snowberger, and Jeff Hemmes
• University of Notre Dame
• http//www.cse.nd.edu/ccl
• IEEE Workshop on Security in Storage 2005

2
Abstract

• Suppose that security in storage has been
  deployed at all endpoints.
• How does this affect the design of distributed
  storage systems that rely upon these devices?
• Clients must become much more
• Fault tolerant, adaptive, and self reliant.
• Aware of resource allocation issues.
• Helpful to the end user!
• Environment Storage Pool at Notre Dame

3
Traditional File System Security
appl
appl
appl
I am John Doe!
untrusted network Ethernet Internet
Security Interface
placement, replication, reliability
File System Abstraction
trusted network PCIRAIDSAN Myrinet Ethernet
disk
disk
disk
Owner of Inode 9842 is UID 56
file
4
Decentralized Security
appl
appl
appl
placement, replication, reliability
Abstr.
Abstr.
Abstr.
untrusted network Ethernet Internet
I am John Doe!
Security
Security
Security
disk
disk
disk
Owner of File /foo/bar is John Doe
file
5
CooperativeStorage SystematNotre Dame
6
What is Cooperative Storage?

• Many devices bound together that can accomplish
  more than one device alone.
• Improve capacity, reliability, performance...
• Could be one person, or many cooperating users.
• Key property
• Each person retains absolute control of their own
  resources by setting local policies.
• People share and collaborate with others that
  they know and trust. No free love! No central
  control!
• However, some resources are set up for the common
  good by an authority. (CS workstations usable by
  any member of the CS department, says the chair.)

7
App
Adapter
???
file system
file system
file system
file system
file system
file system
file system
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
CFS Central File System
appl
appl
appl
adapter
adapter
adapter
CFS
CFS
CFS
file server
file
file
file
15
DSFS Dist. Shared File System
appl
appl
adapter
adapter
DSFS
DSFS
file server
file server
file server
file
file
file
file
file
ptr
file
file
file
file
file
ptr
ptr
16
DSDB Dist. Shared Database
appl
appl
adapter
adapter
DSDB
DSDB
insert
query
direct access
file server
file server
database server
create
file
file
file index
file
file
file
file
file
file
file
file
file
17
Applications

• Simple and Secure Remote Access
• CDF Remote Dynamic Linking
• BaBar Remote Database Access
• LHC Semantic Remote Filesystems
• Distributed File Systems
• GRAND Scalable Archive for Online Data
• Distributed Databases
• GEMS Molecular Dynamics Simulation
• CVRL Biometric Image Storage/Analysis

18
Challenges of Decentralization

• Unbounded Set of Users
• There is no global /etc/passwd or /etc/group!
• Multiple Identities per User
• Kerberos creds from Notre Dame / Wisconsin.
• GSI creds from ND/UW/DOE/NCSA.
• New Decision Points
• Placement decision made, but action fails!
• Directory op succeeds, but file creation fails!
• Unexpected Policy Coupling
• Data placement may affect access control!

19
Outline of Paper

• Centralized vs Decentralized Security
• Architecture of Cooperative Storage
• Basic Security Mechanism
• Problem Complexity Confuses!
• Detail Reservation Right
• Challenges
• Authorization in Distributed File Systems
• Logistics of Third Party Transfer
• Mechanisms for Active Storage
• Semantics of Distributed Group Management

20
Basic Security Mechanism

• Negotiate an Authentication Method
• Client proposes, server agrees/disagrees.
• Default ordering works for most manual
  override.
• Different servers/clients may support diff
  subsets.
• Then, Authenticate via Chosen Method
• May involve challenges, cert exchange, etc...
• Yields a Subject Name for the Session
• kerberosdthain_at_nd.edu
• globus/ONotreDame/CNDouglasThain
• hostnamehedwig.cse.nd.edu
• unixdthain

21
Authorization Mechanism

• Unix Access Controls Are Not Sufficient
• Integer UIDs are not sufficient for principals.
• Nine owner/group/others bits are restrictive.
• Mapping from subjects to Unix is a mess.
• Place Variable Length ACLs on dirs
• globus/ONotreDame/CNDThain RWLAX
• kerberosdthain_at_nd.edu RWL
• hostname.cs.nd.edu
  RL
• globus/ONotreDame/ RL

22
Problem Complexity Confuses!

• For beginning users
• Negotiated authentication makes life easy.
• Everybody can authenticate in some way.
• Most users dont think about it first.
• For advanced users
• Negotiation has unexpected effects.
• What happens when credentials expire?
• For long running / large tasks, better to
  manually specify the authentication mode.
• AuthN failure is easier to retry than authZ
  failure!
• Unexpected authentication is hard to debug.
• Full detail logging mode reveals auth algorithm.
• Always prominently display subject name in all
  tools!

23
Problem Shared Namespace
file server
globus/ONotreDame/ RWLAX
24
Solution Reservation (V) Right
file server
mkdir only!
ONotreDame/CN V(RWLA)
25
Outline of Paper

• Centralized vs Decentralized Security
• Architecture of Cooperative Storage
• Basic Security Mechanism
• Problem Complexity Confuses!
• Detail Reservation Right
• Challenges
• Authorization in Distributed File Systems
• Logistics of Third Party Transfer
• Mechanisms for Active Storage
• Semantics of Distributed Group Management

26
DSFS Dist. Shared File System
appl
appl
adapter
adapter
DSFS
DSFS
file server
file server
file server
file
file
file
file
file
ptr
file
file
file
file
file
ptr
ptr
27
DSFS Logistics

• Consider Creating a File
• Fetch list of resources
• online catalog / static list / user selected
• Make placement decision
• random / fill in order / user selected
• Create stub file on dir server. (fail?)
• Create actual file on data server. (fail?)
• Note that two access controls are in play
• One controls access to the namespace.
• Another controls access to the data storage.

28
DSFS Applications

• Personal Mass Storage
• Expand your local filesystem to include all the
  disks available in a cluster / lab / basement.
• Distributed /tmp for Cluster Computing
• Harness remote cluster for the duration of a job.
• Multi-User Scalable Storage
• Department provides directory, but no space.
• /ONotreDame/OCSE/CN RWL
• Participants provide their own data servers.
• /ONotreDame/OCSE/CNJohnDoe RWLA
• Separates provisioning from access!

29
Dealing with Failure

• Failure to place data is very common!
• Unexpected access controls on device.
• Device is temporarily unavailable. (reboot?)
• Device is newly installed or creds expired.
• Owner changed the sharing policy.
• Soln Client Needs to Model the System
• Track successes and failures on each device.
• Failed devices are not tried again for a time.
• Of course, cannot avoid a device forever...

30
Outline of Paper

• Centralized vs Decentralized Security
• Architecture of Cooperative Storage
• Basic Security Mechanism
• Problem Complexity Confuses!
• Detail Reservation Right
• Challenges
• Authorization in Distributed File Systems
• Logistics of Third Party Transfer
• Mechanisms for Active Storage
• Semantics of Distributed Group Management

31
PINS Processing in Storage

• Observation
• Traditional clusters separate CPU and storage
  into two distinct systems/problems.
• Distributed computing is always some direct
  combination of CPU and I/O needs.
• Idea PINS
• Cluster HW is already a tighly integrated complex
  of CPU and I/O. Make the SW reflect the HW.
• Key Always compute in the same place that the
  data is located. Leave newly created data in
  place.

32
Compute via Passive Storage
Compute YF(X) where XA,B,C,D
F
Y1
Y2
Y3
Y4
file server
file server
file server
file server
(X 200)
A
B
C
D
S1
S2
S3
S4
33
Compute via Active Storage
Compute YF(X) where XA,B,C,D
F
F
F
F
file server
file server
file server
file server
(X 200)
A
B
C
D
S1
S2
S3
S4
34
Technique Identity Boxing
file server
/ directory ACL hostname.cse.nd.edu
RWLX globus/ONotreDame/ RWLX
sim.exe
Identity Box /ONotreDame/CNMonk
in.dat
35
Unified Semantics

• Same Identity for Exec and Data Access
• Stage in data as user X.
• Program runs as user X, data is protected.
• Access results as user X.
• Same ACLs for Exec and Data Access
• Need the X right to run a program.
• RX rights given user can run fixed F(X).
• WX rights given user can stage in any F(X).

36
Outline of Paper

• Centralized vs Decentralized Security
• Architecture of Cooperative Storage
• Basic Security Mechanism
• Problem Complexity Confuses!
• Detail Reservation Right
• Challenges
• Authorization in Distributed File Systems
• Logistics of Third Party Transfer
• Mechanisms for Active Storage
• Semantics of Distributed Group Management

37
Fully Decentralized User Groups

• Distributed Orgs Have Complex Needs
• CMS Collaboration 10s of institutions, 100s of
  PIs, 1000s of graduate students staff.
• There is no centralized database for CMS.
• Local managers add/remove members locally.
• Want Storage Systems that Allow Reference to
  Groups Managed by Others
• Allow access to all staff involved in CMS.
• Allow access to any NSF program manager.
• Allow access to all CS faculty at ND/Purdue.

38
Fully Decentralized ACLs
group lookups
39
Challenges of ACLs

• Performance / Availability / Consistency
• Give the group/ACL owner control.
• Specify maximum time for stale data.
• Implemented, but continuing experience leads to
  reflection on the semantics.
• Example What to do under failures?
• Partial answer servers fail quickly, client
  retries up to a user-controlled limit.
• Consider Group A gives W access, group B gives
  R.
• What happens when group A is unavailable?
• Two very different questions
• What rights does user X have?
• Can user X perform a read?

40
Outline of Paper

• Centralized vs Decentralized Security
• Architecture of Cooperative Storage
• Basic Security Mechanism
• Problem Complexity Confuses!
• Detail Reservation Right
• Challenges
• Authorization in Distributed File Systems
• Logistics of Third Party Transfer
• Mechanisms for Active Storage
• Semantics of Distributed Group Management

41
Practical Lessons

• In a system with decentralized security...
• Users need debugging tools!
• Simple examples whoami, rwhoami
• Client software must become heavier
• Must carefully parse a vast array of errors.
• Must maintain a model of remote devices.
• High level names must be used deep within the
  system software stack.
• Run processes with subject name, not Unix UID.

42
For more information...

• Cooperative Computing Lab
• http//www.cse.nd.edu/ccl
• Cooperative Computing Tools
• http//www.cctools.org
• Douglas Thain
• dthain_at_cse.nd.edu
• http//www.cse.nd.edu/dthain