SCAN: a Scalable, Adaptive, Secure and Network-aware Content Distribution Network Yan Chen CS Department Northwestern University

Motivation The Internet has evolved to become a commercial infrastructure for service delivery –Web delivery, VoIP, streaming media … Challenges for Internet-scale services –Scalability: 600M users, 35M Web sites, 2.1Tb/s –Efficiency: bandwidth, storage, management –Agility: dynamic clients/network/servers –Security: proliferate attacks/viruses/worms E.g., content delivery - Content Distribution Network (CDN) –Web delivery –Grid computing

How CDN Works

Challenges for CDN Content Location –Find nearby replicas with good DoS attack resilience –Dynamic, scalable semantic search Replica Deployment –Dynamics, efficiency –Client QoS (latency, coherence) and server capacity constraints Replica Management –Replica index state maintenance scalability Adaptation to Network Congestion/Failures –Overlay monitoring scalability and accuracy Security –Proactive anomaly/intrusion detection on high-speed network

Provision: Dynamic Replication + Update Multicast Tree Building Replica Management: (Incremental) Content Clustering Network End-to-End Distance Monitoring (latency & loss rate) DHT-based Replica Location: Network DoS Attack Resilient & Semantic Search Support SCAN: Scalable Content Access Network Proactive Anomaly/Intrusion Detection on High-speed Network

Replica Location (security) Existing Work and Problems –Centralized, Replicated and Distributed Directory Services –No security benchmarking, which one has the best DoS attack resilience? Solution –Proposed the first simulation-based network DoS resilience benchmark –Applied it to compare three directory services –DHT-based Distributed Directory Services has best resilience in practice Publication –3 rd Int. Conf. on Info. and Comm. Security (ICICS), 2001

Replica Location (semantic search) Existing Work and Problems –Mostly keyword/title based search –Emerging semantic search systems, but static, unscalable Solution –Apply DHT to distribute the indices –Use “concept indexing” to incrementally grow the semantic space => incrementally add new concepts & documents –Group the indices based on semantic locality => semantic routing, better query accuracy and efficiency

Replica Placement & Coherence Support Existing Work and Problems –Static placement –Dynamic but inefficient placement –No coherence support Solution –Dynamically place close to optimal # of replicas with clients QoS (latency) and servers capacity constraints –Self-organize replica into a scalable application-level multicast for disseminating updates –With overlay network topology only Publication –IPTPS 2002, Pervasive Computing 2002

Existing Work and Problems –Cooperative access for good efficiency requires maintaining replica indices –Per Website replication, scalable, but poor performance –Per URL replication, good performance, but unscalable Solution –Clustering-based replication reduces the overhead significantly without sacrificing much performance –Proposed a unique online Web object popularity prediction scheme based on hyperlink structures –Online incremental clustering and replication to push replicas before accessed Publication –ICNP 2002, IEEE J-SAC 2003 Replica Management

Adaptation to Network Congestion/Failures Existing Work and Problems –Latency estimation systems scalable, but cannot monitor congestion/failures which require n 2 measurement for n end hosts Solution –Tomography-based Overlay Monitoring (TOM) - selectively monitor a basis set of O(n logn) paths to infer the loss rates of other paths –Works in real-time, adapts to topology changes, has good load balancing and tolerates topology errors –Built an adaptive overlay streaming media system on top of TOM –Root-cause diagnosis in progress Publication –Modeling: SIGCOMM IMC 2003 (extended abstract) –Full version under submission

Existing Work and Problems –A/I detection requires flow-level traffic monitoring, unscalable for high-speed network –Most IDS are signature-based, only for known attacks Solution –Leverage “K-ary sketch”, a compact probabilistic summary of flow-level traffic, constant update/query cost, linearity –Use statistical methods, like Hidden Markov Model (HMM) and time series analysis for proactive detection –Profile characteristics of new apps to reduce false positive Publication –K-ary sketch: SIGCOMM IMC 2003 Proactive Anomaly/Intrusion Detection on High-speed Network