The Consequences of Decentralized Security in a Cooperative Storage System Douglas Thain, Chris Moretti, Paul Madrid, Phil Snowberger, and Jeff Hemmes University of Notre Dame IEEE Workshop on Security in Storage 2005

Abstract Suppose that security in storage has been deployed at all endpoints. How does this affect the design of distributed storage systems that rely upon these devices? Clients must become much more: –Fault tolerant, adaptive, and self reliant. –Aware of resource allocation issues. –Helpful to the end user! Environment: Storage Pool at Notre Dame

Traditional File System Security appl Security Interface File System Abstraction disk file Owner of Inode 9842 is UID 56 trusted network: PCI RAID SAN Myrinet Ethernet untrusted network: Ethernet Internet placement, replication, reliability I am John Doe!

Decentralized Security appl Abstr. disk file untrusted network: Ethernet Internet Security Abstr. placement, replication, reliability Owner of File /foo/bar is John Doe I am John Doe!

Cooperative Storage System at Notre Dame

What is Cooperative Storage? Many devices bound together that can accomplish more than one device alone. –Improve capacity, reliability, performance... –Could be one person, or many cooperating users. Key property: –Each person retains absolute control of their own resources by setting local policies. –People share and collaborate with others that they know and trust. No free love! No central control! –However, some resources are set up for the common good by an authority. (CS workstations usable by any member of the CS department, says the chair.)

file transfer file system file system file system file system file system file system file system Central Filesystem App Distributed Database Abstraction Adapter App Distributed Filesystem Abstraction Adapter App Cluster administrator controls policy on all storage in cluster UNIX Workstations owners control policy on each machine. file server file server file server file server file server file server file server UNIX ??? Adapter 3PT

CFS: Central File System file server adapter appl file CFS

ptr DSFS: Dist. Shared File System file server appl file server file server file adapter DSFS lookup file location access data

DSDB: Dist. Shared Database adapter appl file server file server file database server file index query direct access insert create file DSDB

Applications Simple and Secure Remote Access –CDF: Remote Dynamic Linking –BaBar: Remote Database Access –LHC: Semantic Remote Filesystems Distributed File Systems –GRAND: Scalable Archive for Online Data Distributed Databases –GEMS: Molecular Dynamics Simulation –CVRL: Biometric Image Storage/Analysis

Challenges of Decentralization Unbounded Set of Users –There is no global /etc/passwd or /etc/group! Multiple Identities per User –Kerberos creds from Notre Dame / Wisconsin. –GSI creds from ND/UW/DOE/NCSA. New Decision Points –Placement decision made, but action fails! –Directory op succeeds, but file creation fails! Unexpected Policy Coupling –Data placement may affect access control!

Outline of Paper Centralized vs Decentralized Security Architecture of Cooperative Storage Basic Security Mechanism –Problem: Complexity Confuses! –Detail: Reservation Right Challenges –Authorization in Distributed File Systems –Logistics of Third Party Transfer –Mechanisms for Active Storage –Semantics of Distributed Group Management

Basic Security Mechanism Negotiate an Authentication Method –Client proposes, server agrees/disagrees. –Default ordering works for most + manual override. –Different servers/clients may support diff subsets. Then, Authenticate via Chosen Method –May involve challenges, cert exchange, etc... Yields a Subject Name for the Session: –globus:/O=NotreDame/CN=DouglasThain –hostname:hedwig.cse.nd.edu –unix:dthain

Authorization Mechanism Unix Access Controls Are Not Sufficient –Integer UIDs are not sufficient for principals. –Nine owner/group/others bits are restrictive. –Mapping from subjects to Unix is a mess. Place Variable Length ACLs on dirs: globus:/O=NotreDame/CN=DThain RWLAX RWL hostname:*.cs.nd.edu RL globus:/O=NotreDame/* RL

Problem: Complexity Confuses! For beginning users: –Negotiated authentication makes life easy. –Everybody can authenticate in some way. –Most users don’t think about it first. For advanced users: –Negotiation has unexpected effects. –What happens when credentials expire? –For long running / large tasks, better to manually specify the authentication mode. –AuthN failure is easier to retry than authZ failure! Unexpected authentication is hard to debug. –Full detail logging mode reveals auth algorithm. –Always prominently display subject name in all tools!

Problem: Shared Namespace file server globus:/O=NotreDame/* RWLAX a.out test.ctest.dat cms.exe

Solution: Reservation (V) Right file server O=NotreDame/CN=* V(RWLA) /O=NotreDame/CN=Monk RWLA mkdir a.outtest.c /O=NotreDame/CN=Monk mkdir /O=NotreDame/CN=Ted RWLA a.outtest.c /O=NotreDame/CN=Ted mkdir only!

Outline of Paper Centralized vs Decentralized Security Architecture of Cooperative Storage Basic Security Mechanism –Problem: Complexity Confuses! –Detail: Reservation Right Challenges –Authorization in Distributed File Systems –Logistics of Third Party Transfer –Mechanisms for Active Storage –Semantics of Distributed Group Management

ptr DSFS: Dist. Shared File System file server appl file server file server file adapter DSFS lookup file location access data

DSFS Logistics Consider Creating a File: –Fetch list of resources: online catalog / static list / user selected –Make placement decision: random / fill in order / user selected –Create stub file on dir server. (fail?) –Create actual file on data server. (fail?) Note that two access controls are in play: –One controls access to the namespace. –Another controls access to the data storage.

DSFS Applications Personal Mass Storage –Expand your local filesystem to include all the disks available in a cluster / lab / basement. Distributed /tmp for Cluster Computing –Harness remote cluster for the duration of a job. Multi-User Scalable Storage –Department provides directory, but no space. /O=NotreDame/O=CSE/CN=* RWL –Participants provide their own data servers. /O=NotreDame/O=CSE/CN=JohnDoe RWLA –Separates provisioning from access!

Dealing with Failure Failure to place data is very common! –Unexpected access controls on device. –Device is temporarily unavailable. (reboot?) –Device is newly installed or creds expired. –Owner changed the sharing policy. Soln: Client Needs to Model the System –Track successes and failures on each device. –Failed devices are not tried again for a time. –Of course, cannot avoid a device forever...

Outline of Paper Centralized vs Decentralized Security Architecture of Cooperative Storage Basic Security Mechanism –Problem: Complexity Confuses! –Detail: Reservation Right Challenges –Authorization in Distributed File Systems –Logistics of Third Party Transfer –Mechanisms for Active Storage –Semantics of Distributed Group Management

PINS: Processing in Storage Observation: –Traditional clusters separate CPU and storage into two distinct systems/problems. –Distributed computing is always some direct combination of CPU and I/O needs. Idea: PINS –Cluster HW is already a tighly integrated complex of CPU and I/O. Make the SW reflect the HW. –Key: Always compute in the same place that the data is located. Leave newly created data in place.

Compute via Passive Storage file server ABCD (X 200) S1 S2S3S4 Compute Y=F(X) where X={A,B,C,D} F Y1Y2Y3Y4

Compute via Active Storage file server ABCD (X 200) S1 S2S3S4 Compute Y=F(X) where X={A,B,C,D} F Y1Y2Y3Y4 FFF

Technique: Identity Boxing / directory ACL: hostname:*.cse.nd.edu RWLX globus:/O=NotreDame/* RWLX file server sim.exe storage owner in.dat Identity Box: /O=NotreDame/CN=Monk sim.exe out.dat client > open x.nd.edu > put sim.exe > put in.dat > exec sim.exe > get out.dat

Unified Semantics Same Identity for Exec and Data Access –Stage in data as user X. –Program runs as user X, data is protected. –Access results as user X. Same ACLs for Exec and Data Access –Need the X right to run a program. –RX rights – given user can run fixed F(X). –WX rights – given user can stage in any F(X).

Outline of Paper Centralized vs Decentralized Security Architecture of Cooperative Storage Basic Security Mechanism –Problem: Complexity Confuses! –Detail: Reservation Right Challenges –Authorization in Distributed File Systems –Logistics of Third Party Transfer –Mechanisms for Active Storage –Semantics of Distributed Group Management

Fully Decentralized User Groups Distributed Orgs Have Complex Needs –CMS Collaboration: 10s of institutions, 100s of PIs, 1000s of graduate students staff. –There is no centralized database for CMS. –Local managers add/remove members locally. Want Storage Systems that Allow Reference to Groups Managed by Others: –Allow access to all staff involved in CMS. –Allow access to any NSF program manager. –Allow access to all CS faculty at ND/Purdue.

Fully Decentralized ACLs univ.edu members univ.edu members file system file server file client read Access Control List group:ccl.nd.edu/faculty RWL group:serv.nsf.gov/managers RL group:ftp.cern.org/members RL check ACL ccl.nd.eduserv.nsf.gov facultymanagers ftp.cern.org members group lookups group lookups

Challenges of ACLs Performance / Availability / Consistency –Give the group/ACL owner control. –Specify maximum time for stale data. Implemented, but continuing experience leads to reflection on the semantics. Example: What to do under failures? –Partial answer: servers fail quickly, client retries up to a user-controlled limit. –Consider: Group A gives W access, group B gives R. –What happens when group A is unavailable? –Two very different questions: What rights does user X have? Can user X perform a read?

Outline of Paper Centralized vs Decentralized Security Architecture of Cooperative Storage Basic Security Mechanism –Problem: Complexity Confuses! –Detail: Reservation Right Challenges –Authorization in Distributed File Systems –Logistics of Third Party Transfer –Mechanisms for Active Storage –Semantics of Distributed Group Management

Practical Lessons In a system with decentralized security... Users need debugging tools! –Simple examples: whoami, rwhoami Client software must become “heavier” –Must carefully parse a vast array of errors. –Must maintain a model of remote devices. High level names must be used deep within the system software stack. –Run processes with subject name, not Unix UID.

For more information... Cooperative Computing Lab Cooperative Computing Lab Cooperative Computing Tools Cooperative Computing Tools Douglas Thain Douglas Thain –