Self-service, with applications to distributed classifier construction Michael K. Reiter and Asad Samar April 27, 2006 Properties & Related Work Self-Service achieves three properties Scalability Single clusters are only incrementally scalable Consistency Most P2P systems do not achieve strong consistency properties Multiple clusters (one per object) lack consistency for multi-object operations Failure recovery DHTs supporting atomic operations do not recover from failures Self-Service recovers from failures but may lose recent operations Consistency Scalability Fault recovery P2P systems [read-only, eventual consistency] Single clusters (Dynamic CDNs) Self-Service Distributed directories, DHT with atomic operations Multiple clusters (e.g., Oceanstore) Self-Service is a different point in this design space Service ImplementationsOur approach Centralized implementation Centralized services Provides strong consistency, but Scales poorly Cluster implementation Cluster-based services Can provide strong consistency, but Requires adding resources to the cluster as number of clients increases Self-service Scales to a large client population (particularly compute scalability) Does not require adding resources to the server as clients increase Provides strong consistency guarantees Can recover from client failures … but best suited for services where Clients are trustworthy (ongoing work) and churn is low Service state can be decomposed into small objects such that client operations typically involve only a few The Self-Service approach Clients perform their own operations No resource increase required at server as clients increase Algorithms guarantee strong consistency properties Can recover from client failures, though at some cost Clients are arranged in a tree rooted at server Service state divided into small objects Each client retrieves the objects required for its operation And applies the operation locally ababababab `a’ gets the object from root and applies its operation, modifying the object. `b’ gets object from `a’ and applies its operation, modifying the object again. Algorithms Nodes arrange themselves in a logical distributed queue And take turns to apply their operations on the object a b c d e abcdabcdebcde a b c d e a b c d e b has requested from a, c from b and d from c. e now requests from d. a completes operation and transfers to b. If a node disconnects, any objects in its sub-tree are reconstituted from latest state seen at any connected node Connected nodes continue to make progress even if a node in the distributed queue gets disconnected p' p c a b d Object history p' p c a b d Object history Object is in the disconnected sub-tree Object reconstituted at the parent of disconnected node Motivation Building models for network traffic classification and intrusion detection From traffic seen at multiple vantage points Globally share information to build accurate models Centralized scenario Send raw data to server Server updates model Server makes new model available Scalability and privacy concerns Our “self-service” approach All participating networks placed in a distributed tree Participating network pulls model Participating network updates model Participating network makes new model available Scales well and raw data not shared with other networks Experiments Implemented algorithms and traffic modeling application Micro-benchmarks on PlanetLab Model building experiments (CPU intensive) on local cluster Compared with a centralized implementation of same service