1. After the Internet Ken Birman Professor, Dept. of Computer Science Cornell University

2. My Background in CIP Professor at Cornell since 1982: 19 years of research on reliable, secure communications software. Author of 150 papers, 2 books… founder and CEO of two companies My software is used by New York and Swiss Stock Exchanges, French air traffic control system, AEGIS warship Lead 1995 DARPA ISAT study of Critical Infrastructure Assurance, recommendations used to retarget DARPA ITO programs

3. Critical Infrastructure Rapidly Expanding Web of Dependency Massive rollout underway Control of restructured power grid New medical information systems link hospital to other providers, reach right into the home Telephony infrastructure Financial systems: eMoney replaces cash! Disaster response and coordination Future military will be extremely dependent on information resources and solutions

4. Tangled Interdependencies Power Grid Internet Telephony Banking Internet Software, COTS Technology Base

5. Multiple Concerns Infrastructure industries have been dangerously naïve about challenges of using Internet and computing technologies in critical ways Nationally critical information systems poorly protected, fragile, easily disrupted Stems from pervasive use of COTS components Vendors poorly motivated to address the issue Yet academic research is having little impact No sense of “excitement” or importance Few significant technology transition successes

6. Most serious issue? Loss of public interest and enthusiasm Government shares this view “It’s just software; we buy it from Microsoft” Academic researchers often seen as freeloading at taxpayer’s expense Critical infrastructure components often look “less critical” considered in isolation Ten thousand networked medical care systems would worry us, but not individual instances

7. Concrete Examples of Threats? Power system requires new generation of technology for preventing cascaded failures, implementing load-following power contracts Industry requires solutions but has no idea how to build them. Technical concern “masked” by politics DOE effort is completely inadequate Three branches of military are separately developing real-time information support tools. Scale will be orders of magnitude beyond anything ever done with Internet technologies Goals recall the FAA’s AAS fiasco (lost $6B!)

8. Vendor Perspective? Little interest in better security “You have zero privacy anyway. Get over it.” Scott McNealy, CEO Sun Microsystems; 1/99 Gates recently suggested that perhaps MSFT needs to improve, but doesn’t have critical infrastructure in mind and didn’t point to Internet issues. Internet technology is adequate for the most commercially lucrative Web functions But inadequate reliability, security for other emerging needs, including CIP requirements Issue is that market is the main driver for product evolution, and market for critical solutions is small

9. Security: Often mistaken for the whole story Even today, most CIP work emphasizes security and denial of service attacks But critical applications must also work Correctly When and where required Even when components fail or are overloaded Even when the network size grows or the application itself is used on a large scale Even when the network is disrupted by failures

10. Let’s get technical A digression to illustrate both the potential for progress but also the obstacles we confront!

11. Scalability: Achilles Heel of a Networked World? 1980’s: Client-server architectures. 1 server, 10’s of simultaneous clients 1990’s: Web servers Small server cluster in a data center or farm 1000’s of simultaneous clients First decade of 2000? Server “geoplex”: large farms in a WAN setting 10’s of 1000’s of simultaneous clients Emergence of peer-to-peer applications: “live” collaboration and sharing of objects Wireless clients could add another factor of 10 client load

12. Technologies need to keep pace We want predictable, stable performance, reliability, security … despite Large numbers of users Large physical extent of network Increasing rates of infrastructure disruption (purely because of growing span of network) Wide range of performance profiles Growth in actual volume of work applications are being asked to do

13. Scalable Publish Subscribe A popular paradigm; we’ll use it to illustrate our points Used to link large numbers of information sources in commercial or military settings to even larger numbers of consumers Track down the right servers Updates in real-time as data changes Happens to be a top military priority, so one could imagine the government tackling it…

14. Server cluster Subscriber must identify the best servers. Subjects are partitioned among servers hence one subscriber may need multiple connections Publisher offers new events to a proxy server. Subjects are partitioned among the server sets. In this example there are four partitions: blue, green, yellow and red. Server set and partition function can adjust dynamically Like the subscribers, each publisher connects to the “best” proxy (or proxies) given its own location in the network. The one selected must belong to the partition handling the subject of the event. log publish

15. Large-scale applications with similar technical requirements Restructured Electric Power Grid Large-scale financial applications Disaster response Community medical systems Large-scale online information provision Decentralized stock markets Network monitoring, control

16. Poor Scalability Long “rumored” for distributed computing technologies and tools Famous study by Jim Gray points to scalability issues in distributed databases Things that scale well: Tend to be stateless or based on soft state Have weak reliability semantics Are loosely coupled

17. Do current technologies scale? CategoryTypical large useLimits? Client-Server and object- oriented environments LAN system, perhaps 250 simultaneous clients Server capacity limits scale. Web-like architecturesInternet, hundreds of clients No reliability guarantees Publish-subscribe Group multicast About 50 receivers, 500 in hierarchies Throughput becomes unstable with scale. Multicast storms Many-Many DSMRarely seen except in small clusters Update costs grow with cluster size Shared databaseFarm: 50-100; RACS: 100’s, RAPS: 10’s Few successes with rapidly changing real-time data

18. Stock Exchange Problem: Vsync. multicast is too “fragile” Most members are healthy…. … but one is slow Most members are healthy….

19. With 32 processes…. 00.10.20.30.40.50.60.70.80.9 0 50 100 150 200 250 Virtually synchronous Ensemble multicast protocols perturb rate average throughput on nonperturbed members ideal actual

20. The problem gets worse as the system scales up 00.10.20.30.40.50.60.70.80.9 0 50 100 150 200 250 Virtually synchronous Ensemble multicast protocols perturb rate average throughput on nonperturbed members group size: 32 group size: 64 group size: 96 32 96

21. Why doesn’t anything scale? With weak semantics… Faulty behavior may occur more often as system size increases (think “the Internet”) With strong semantics… Encounter a system-wide cost (e.g. membership reconfiguration, congestion control) That can be triggered more often as a function of scale (more failures, or more network “events”, or bigger latencies) Gray’s O(n 2 ) database degradation reflects very similar issues… a new law of nature?

22. Serious issue for our scalable publish-subscribe technology What if we build it for the military or some other critical use, and it works in the laboratory but not in the field? Early evaluation has ruled out most off-the-shelf networking technologies They just don’t have the necessary scalability! In fact, this happened with Navy’s Cooperative Engagement Capability (CEC) They built it… but it melts down under stress!

23. Fight fire with fire! Turn to randomized protocols… … with probabilistic reliability goals This overcomes the scalability problems just seen Then think about how to “present” mechanism to user

24. Cornell Scalability Research Spinglass: Scalable technologies that step into the gap Includes two major components: Astrolabe: A resource location technology Bimodal Multicast: A scalable multicast

25. Astrolabe Goal is to create a dynamic database showing continuously evolving state of the programs comprising some system We’ll use this to build better systems Approach: “peer to peer gossip”. Basically, each machine has a piece of a jigsaw puzzle. Assemble it on the fly

26. Astrolabe in a single domain NameLoadWeblogic?SMTP?Word Version… swift2.0016.2 falcon1.5104.1 cardinal4.5106.0 Row can have many columns Total size should be k-bytes, not megabytes Configuration certificate determines what data is pulled into the table (and can change) 3.1 5.3 0.9 1.9 3.6 0.8 2.1 2.7 1.1 1.8

27. Build a hierarchy using a P2P protocol that “assembles the puzzle” without any servers NameLoadWeblogic?SMTP?Word Version … swift2.0016.2 falcon1.5104.1 cardinal4.5106.0 NameLoadWeblogic?SMTP?Word Version … gazelle1.7004.5 zebra3.2016.2 gnu.5106.2 NameAvg Load WL contactSMTP contact SF2.6123.45.61.3123.45.61.17 NJ1.8127.16.77.6127.16.77.11 Paris3.114.66.71.814.66.71.12 San Francisco New Jersey SQL query “summarizes” data Dynamically changing query output is visible system-wide

28. (1) Query goes out… (2) Compute locally… (3) results flow to top level of the hierarchy NameLoadWeblogic?SMTP?Word Version … swift2.0016.2 falcon1.5104.1 cardinal4.5106.0 NameLoadWeblogic?SMTP?Word Version … gazelle1.7004.5 zebra3.2016.2 gnu.5106.2 NameAvg Load WL contactSMTP contact SF2.6123.45.61.3123.45.61.17 NJ1.8127.16.77.6127.16.77.11 Paris3.114.66.71.814.66.71.12 San Francisco New Jersey 1 33 1 2 2

29. Hierarchy is virtual… data is replicated NameLoadWeblogic?SMTP?Word Version … swift2.0016.2 falcon1.5104.1 cardinal4.5106.0 NameLoadWeblogic?SMTP?Word Version … gazelle1.7004.5 zebra3.2016.2 gnu.5106.2 NameAvg Load WL contactSMTP contact SF2.6123.45.61.3123.45.61.17 NJ1.8127.16.77.6127.16.77.11 Paris3.114.66.71.814.66.71.12 San Francisco New Jersey

30. Hierarchy is virtual… data is replicated NameLoadWeblogic?SMTP?Word Version … swift2.0016.2 falcon1.5104.1 cardinal4.5106.0 NameLoadWeblogic?SMTP?Word Version … gazelle1.7004.5 zebra3.2016.2 gnu.5106.2 NameAvg Load WL contactSMTP contact SF2.6123.45.61.3123.45.61.17 NJ1.8127.16.77.6127.16.77.11 Paris3.114.66.71.814.66.71.12 San Francisco New Jersey

31. Examples? A flexible, user-programmable mechanism Which sensors are reporting detection of low- levels of chemical warfare agents? Which soldiers are downwind from location X,Y? Where can I find intelligence about the building located at coordinates X,Y? Which machines are running WebLogic v 3.2? Think of aggregation functions as small agents that look for information When changes occur, aggregated table reflects those changes within seconds

32. Astrolabe summary Scalable: technology can support hundreds of thousands of participants Flexible: can easily extend domain hierarchy, define new columns or eliminate old ones Secure: Uses keys for authentication and can even encrypt Handles firewalls gracefully, including issues of IP address re-use behind firewalls Performs well: updates propagate in seconds Cheap to run: tiny load, small memory impact

33. Contrast with most P2P schemes Our peer-to-peer approach is implemented using pseudo-random gossip In contrast most peer-to-peer architectures Are specifically intended to support file systems Don’t use pseudo-random P2P patterns Any hierarchical structure is “real”; ours is an abstraction constructed by the protocol itself

34. Bimodal Multicast Work we did before developing Astrolabe A multicast, hence more directly comparable with older replication technology Also uses peer-to-peer gossip. We’ll review it to illustrate the approach

35. Reminder: Multicast scaling issue 00.10.20.30.40.50.60.70.80.9 0 50 100 150 200 250 Virtually synchronous Ensemble multicast protocols perturb rate average throughput on nonperturbed members group size: 32 group size: 64 group size: 96 32 96

36. Start by using unreliable multicast to rapidly distribute the message. But some messages may not get through, and some processes may be faulty. So initial state involves partial distribution of multicast(s) Uses IP multicast (unreliable) to distribute data

37. Periodic gossip used to repair message loss. (In reality, it isn’t synchronized, and we do all sorts of things to avoid excessive gossip over WAN links or slow connections) Rounds of gossip repair gaps

38. Bimodal Multicast uses gossip Dates back to NNTP, Clearinghouse. Best papers are by Demers et. al. Periodically, each process picks some other process, merge states Mathematics: epidemic infection Must tune to deal with bandwidth, latency issues, other pragmatics For example, if a region is missing some data, we re-multicast it locally

39. Figure 5: Graphs of analytical results Bimodal Multicast is amenable to formal analysis

41. Unlimited scalability! Probabilistic gossip “routes around” congestion And probabilistic reliability model lets the system move on if a computer lags behind Results in: Constant communication costs Constant loads on links Steady behavior even under stress

42. Server cluster Subscriber must identify the best servers. Subjects are partitioned among servers hence one subscriber may need multiple connections Publisher offers new events to a proxy server. Subjects are partitioned among the server sets. In this example there are four partitions: blue, green, yellow and red. Server set and partition function can adjust dynamically Like the subscribers, each publisher connects to the “best” proxy (or proxies) given its own location in the network. The one selected must belong to the partition handling the subject of the event. log publish

43. Server cluster Subscriber must identify the best servers. Publisher uses Astrolabe to identify the correct set of receivers log Bimodal Multicast Astrolabe manages configuration and connection parameters, tracks system membership and state. The combined technologies solve the initial problem!

44. Good things? Both technologies overcome Internet limitations using randomized P2P gossip However, Internet routing can “defeat” our clever solutions unless we know network topology Both have great scalability and can survive under stress And both are backed by formal models as well as real code and experimental data Indeed, analysis is “robust” too!

45. Bad things? These are middleware, and the bottom line is that only MSFT can sell middleware! Current commercial slump doesn’t help; nobody is buying anything Indeed, while everything else advances at “Internet speed”… the Internet architecture has somehow gotten stuck circa 1985! The Internet: Unsafe at any speed?

46. The Internet “policy” Assumes almost everything uses TCP TCP is designed to be greedy Ratchet bandwidth up until congestion occurs Routers are designed to drop packets They use RED (Random Early Detection) Throw away packets at random until TCP gets the point and slows down Our problem? We’re not running TCP and this policy penalizes us, although it “works” for TCP….

47. Internet itself: Main weak point Our hardest open problems arise in the Internet Astrolabe and Bimodal Multicast don’t do much for security They need to know network topology… but the Internet conceals this information We could perhaps use these tools to detect and react to a DOS attack at the application layer, but in fact such an attack can only be stopped in the network itself Butler Lampson: “The Internet and the Web are successful precisely because they don’t need to work very well to succeed”

48. The Internet got stuck in 1985 Critical Infrastructure Protection hostage to a perception that the Internet is perfect! Must somehow recapture the enthusiasm of the field and the commercial sector for evolution and change Scalability: building massive systems that work really well and yet make full use of COTS Awesome performance, even under stress Better Internet: Time for a “Supernet”?

49. Lagging public interest An extremely serious problem The Internet boomed… then it melted down And we’re Internet people Even worse in the CIP area We predicted disaster in 1996… 1999… 2000 Cyberterrorists… “Internet will melt down” We’re the people who keep crying wolf Realistically, can’t fight this perception Argues that CIP success will have to come from other pressures, not a direct public clamor!

50. Best hope? Government could require that suppliers implement best practice security and “defense” techniques Will MSFT emphasis on reliability and security trigger a wave of commercial products in these areas? Or will the 800lb gorilla just crush the whole market? Reexamine legal basis for “hold harmless” clauses that indemnify software vendors against damages if products are defective through outright negligence Growing need for homeland defense tools might help

51. Conclusions CIP hostage to complacency as an undramatic threat slowly grows! Nationally critical infrastructure is exposed to security & reliability problems and this exposure is growing, yet is largely ignored. Research effort has contracted around an overly theoretical security community Current trend a recipe for economic stagnation. Inadequate technology blocks new markets