1 The Endeavour Expedition: 21st Century Computing to the eXtreme Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley.

Slides:



Advertisements
Similar presentations
2  Industry trends and challenges  Windows Server 2012: Modern workstyle, enabled  Access from virtually anywhere, any device  Full Windows experience.
Advertisements

 What Is Desktop Virtualization?  How Does Application Virtualization Help?  How does V3 Systems help?  Getting Started AGENDA.
Current impacts of cloud migration on broadband network operations and businesses David Sterling Partner, i 3 m 3 Solutions.
High Performance Computing Course Notes Grid Computing.
The future of Desktops Transform Your Desktop with Virtualization.
1 The Endeavour Expedition: 21st Century Computing to the eXtreme Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley.
A Java Architecture for the Internet of Things Noel Poore, Architect Pete St. Pierre, Product Manager Java Platform Group, Internet of Things September.
Don’t Let Anybody Slip into Your Network! Using the Login People Multi-Factor Authentication Server Means No Tokens, No OTP, No SMS, No Certificates MICROSOFT.
® IBM India Research Lab © 2006 IBM Corporation Challenges in Building a Strategic Information Integration Infrastructure Mukesh Mohania IBM India Research.
Option 2: The Oceanic Data Utility: Global-Scale Persistent Storage John Kubiatowicz.
System Architecture for Extreme Devices David Culler U.C. Berkeley DARPA Meeting 9/21/1999.
1 The Endeavour Expedition: Computing and Communications at the eXtremes Professor Randy H. Katz IBM Research Visit 15 June Soda Hall.
1 Internet-Scale Systems Research Group (ISRG) Eric Brewer, David Culler, Anthony Joseph, Randy Katz Computer Science Division, EECS Department University.
1 Internet-Scale Systems Research Group (ISRG) Eric Brewer, David Culler, Anthony Joseph, Randy Katz Computer Science Division, EECS Department University.
Information Capture and Re-Use Joe Hellerstein. Scenario Ubiquitous computing is more than clients! –sensors and their data feeds are key –smart dust.
1 The Endeavour Expedition: Charting the Fluid Information Utility Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley.
Think. Learn. Succeed. Aura: An Architectural Framework for User Mobility in Ubiquitous Computing Environments Presented by: Ashirvad Naik April 20, 2010.
Option 2: The Oceanic Data Utility: Global-Scale Persistent Storage John Kubiatowicz.
Extensible Scalable Monitoring for Clusters of Computers Eric Anderson U.C. Berkeley Summer 1997 NOW Retreat.
1 The Endeavour Expedition: Charting the Fluid Information Utility Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley.
1 The Endeavour Expedition: Computing and Communications at the eXtremes Professor Randy H. Katz CS Division, EECS Department University of California,
Integrated Scientific Workflow Management for the Emulab Network Testbed Eric Eide, Leigh Stoller, Tim Stack, Juliana Freire, and Jay Lepreau and Jay Lepreau.
Operating System Architecture for Incredibly Diverse Devices David Culler U.C. Berkeley Expeditions Meeting 8/16/1999.
Finale’ cs294-8 Design of Deeply Networked Systems Spring 2000 David Culler & Randy Katz U.C. Berkeley
ProActive Infrastructure Eric Brewer, David Culler, Anthony Joseph, Randy Katz Computer Science Division U.C. Berkeley ninja.cs.berkeley.edu Active Networks.
Telegraph: An Adaptive Global- Scale Query Engine Joe Hellerstein.
The Oceanic Data Utility: (OceanStore) Global-Scale Persistent Storage John Kubiatowicz.
Connecting the Invisible Extremes of Computing David Culler U.C. Berkeley Summer Inst. on Invisible Computing July,
1 The Endeavour Expedition: 21st Century Computing to the eXtreme Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley.
1 The Endeavour Expedition: 21st Century Computing to the eXtreme Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley.
1 Internet-Scale Systems Research Group (ISRG) Retreat Winter 2000 Eric Brewer, David Culler, Anthony Joseph, Randy Katz Computer Science Division, EECS.
Clusters Massive Cluster Gigabit Ethernet System Design for Vastly Diverse Devices David Culler U.C. Berkeley HP Visit 3/9/2000.
1 Internet-Scale Systems Research Group Eric Brewer, David Culler, Anthony Joseph, Randy Katz, Steven McCanne Computer Science Division, EECS Department.
Little Demonstration of the Power in Discovery Jason Hill, Steve Ross David E. Culler Computer Science Division U.C. Berkeley.
1 Internet-Scale Systems Research Group (ISRG) Eric Brewer, David Culler, Anthony Joseph, Randy Katz, Steven McCanne Computer Science Division, EECS Department.
1 Reliable Adaptive Distributed Systems Armando Fox, Michael Jordan, Randy H. Katz, David Patterson, George Necula, Ion Stoica, Doug Tygar.
Universal Berkeley Activities in the ISRG / Endeavour David Culler Randy Katz, Eric Brewer, Anthony Joseph, James Landay and others
Packing for the Expedition David Culler. 5/25/992 Ongoing Endeavors Millennium: building a large distributed experimental testbed –Berkeley Cluster Software.
Internet-Scale Systems Research Group Eric Brewer David Culler Anthony Joseph Randy Katz Steven McCanne Computer Science Division University of California,
Effectively and Securely Using the Cloud Computing Paradigm.
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
Hosted on the Powerful Microsoft Azure Platform, Advent Countdown Lets Companies Run Reliable and Scalable Holiday Marketing Campaigns MICROSOFT AZURE.
Cloud Computing. What is Cloud Computing? Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable.
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved DISTRIBUTED.
An emerging computing paradigm where data and services reside in massively scalable data centers and can be ubiquitously accessed from any connected devices.
Copyright © 2011 EMC Corporation. All Rights Reserved. MODULE – 6 VIRTUALIZED DATA CENTER – DESKTOP AND APPLICATION 1.
Chapter © 2012 Pearson Education, Inc. Publishing as Prentice Hall.
BUSINESS DRIVEN TECHNOLOGY
Virtual Data Grid Architecture Ewa Deelman, Ian Foster, Carl Kesselman, Miron Livny.
Copyright © 2002 Intel Corporation. Intel Labs Towards Balanced Computing Weaving Peer-to-Peer Technologies into the Fabric of Computing over the Net Presented.
1 Evolution and Revolution: Windows 7 and Desktop Virtualization How to Accelerate Migration to Windows 7 Miguel Sian, Sr. Enterprise Solutions Consultant.
Introduction Infrastructure for pervasive computing has many challenges: 1)pervasive computing is a large aspect which includes hardware side (mobile phones,portable.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Securely Synchronize and Share Enterprise Files across Desktops, Web, and Mobile with EasiShare on the Powerful Microsoft Azure Cloud Platform MICROSOFT.
ProActive Infrastructure Eric Brewer, David Culler, Anthony Joseph, Randy Katz Computer Science Division U.C. Berkeley ninja.cs.berkeley.edu Active Networks.
Internet of Things. IoT Novel paradigm – Rapidly gaining ground in the wireless scenario Basic idea – Pervasive presence around us a variety of things.
To provide the world with a next generation storage platform for unstructured data, enabling deployment of mobile applications, virtualization solutions,
Ellis Paul Technical Solution Specialist – System Center Microsoft UK Operations Manager Overview.
Societal-Scale Computing: The eXtremes Scalable, Available Internet Services Information Appliances Client Server Clusters Massive Cluster Gigabit Ethernet.
Axis AI Solves Challenges of Complex Data Extraction and Document Classification through Advanced Natural Language Processing and Machine Learning MICROSOFT.
Built on the Powerful Microsoft Azure Platform, Forensic Advantage Helps Public Safety and National Security Agencies Collect, Analyze, Report, and Distribute.
© 2012 Eucalyptus Systems, Inc. Cloud Computing Introduction Eucalyptus Education Services 2.
Enterprise Alert on Microsoft Azure Fully Automates Critical Incident Communication and Transforms It into an Intelligent, Reliable, and Mobile Experience.
Grid Services for Digital Archive Tao-Sheng Chen Academia Sinica Computing Centre
Clouds , Grids and Clusters
Option 2: The Oceanic Data Utility: Global-Scale Persistent Storage
Internet-Scale Systems Research Group
Telegraph: An Adaptive Global-Scale Query Engine
The Endeavour Expedition: Charting the Fluid Information Utility
Information Capture and Re-Use
Presentation transcript:

1 The Endeavour Expedition: 21st Century Computing to the eXtreme Randy H. Katz, Principal Investigator EECS Department University of California, Berkeley Berkeley, CA

2 The Endeavour Expedition: 21st Century Computing to the eXtreme R. H. Katz, Principal Investigator, University of California, Berkeley New Ideas Systems Architecture for Vastly Diverse Computing Devices (MEMS, cameras, displays) Wide-area “Oceanic” Data Information Utility Sensor-Centric Data Management for Capture and Reuse (MEMS + networked storage) Negotiation Architecture for Cooperating Components (Composable system architecture) Tacit Knowledge Infrastructure to support High-Speed Decision-Making Information Management for Intelligent Classroom Environments Scalable Safe Component-based Design and UI Design Tools Impact Enhancing human understanding by making it dramatically more convenient for people to interact with information, devices, and other people Supported by a “planetary-scale” Information Utility, stress tested by applications in decision making and learning, achieved thru new methodologies for design, construction, and administration of systems of unprecedented scale and complexity Schedule Jun 99 Start Jun 00Jun 01May 02 End Initial Architectural Design & Testbeds Initial Application Implementation & Evaluation Information Utility Information Applications Design Methodologies Initial Evaluation & 2nd Gen Redesign Final Deployment & Evaluation Refined Implementation & Final Evaluation Usability Studies & Early Tool Design Implementation of UI &Sys Design Tools Tools Release & Final Evaluations Initial Architectural Design Document Initial Experiments & Revised Design Doc Final Experiments & Architecture Docs

3 Expedition Goals Enhancing understanding –Dramatically more convenient for people to interact with information, devices, and other people –Supported by a “planetary-scale” Information Utility »Stress tested by challenging applications in decision making and learning »New methodologies for design, construction, and administration of systems of unprecedented scale and complexity –Figure of merit: how effectively we amplify and leverage human intellect A pervasive Information Utility, based on “fluid systems” to enable new approaches for problem solving & learning

4 Why “Endeavour”? Endeavour: to strive or reach; a serious determined effort (Webster’s 7th New Collegiate Dictionary); British spelling Captain Cook’s ship from his first voyage of exploration of the great unknown of his day: the southern Pacific Ocean ( ) –Brought more land and wealth to the British Empire than any military campaign –Cook’s lasting contribution: comprehensive knowledge of the people, customs, and ideas that lay across the sea –“He left nothing to his successors other than to marvel at the completeness of his work.”

5 Expedition Assumptions Human time and attention, not processing or storage, are the limiting factors Givens: –Vast diversity of computing devices (PDAs, cameras, displays, sensors, actuators, mobile robots, vehicles); No such thing as an “average” device –Unlimited storage: everything that can be captured, digitized, and stored, will be –Every computing device is connected in proportion to its capacity –Devices are predominately compatible rather than incompatible (plug-and-play enabled by on-the-fly translation/adaptation)

6 Expedition Challenges Personal Information Mgmt is the Killer App –Not corporate processing but management, analysis, aggregation, dissemination, filtering for the individual People Create Knowledge, not Data –Not management/retrieval of explicitly entered information, but automated extraction and organization of daily activities Information Technology as a Utility –Continuous service delivery, on a planetary-scale, on top of a highly dynamic information base Beyond the Desktop –Community computing: infer relationships among information, delegate control, establish authority

7 Driving Factors Technology Push –Accelerating developments at the eXtremes: »Cluster-based compute/storage servers »MEMS sensor/actuators, CCD cameras, LCD displays, … User Pull –More effective community leverage: the next power tool –Desire: »Enhanced interaction, ease of use »Easier configuration, “plug and play” »Less fragile tools, “always there” utility functionality

8 Computing Revolution: Devices in the eXtreme Evolution Information Appliances: Scaled down desktops, e.g., CarPC, PdaPC, etc. Evolved Desktops Servers: Scaled-up Desktops, Millennium Revolution Information Appliances: Many computers per person, MEMs, CCDs, LCDs, connectivity Servers: Integrated with comms infrastructure; Lots of computing in small footprint Display Keyboard Disk Mem  Proc PC Evolution Display Camera Smart Sensors Camera Smart Spaces Computing Revolution WAN Server, Mem, Disk Information Utility BANG! Display Mem Disk  Proc

9 Expedition Approach Information Devices –Beyond desktop computers to MEMS-sensors/actuators with capture/display to yield enhanced activity spaces Information Utility Information Applications –High Speed/Collaborative Decision Making and Learning –Augmented “Smart” Spaces: Rooms and Vehicles Design Methodology –User-centric Design with HW/SW Co-design; –Formal methods for safe and trustworthy decomposable and reusable components “Fluid”, Network-Centric System Software –Partitioning and management of state between soft and persistent state –Data processing placement and movement –Component discovery and negotiation –Flexible capture, self- organization, and re-use of information

10 Interdisciplinary, Technology- Centered Expedition Team Alex Aiken, PL Eric Brewer, OS John Canny, AI David Culler, OS/Arch Joseph Hellerstein, DB Michael Jordan, Learning Anthony Joseph, OS Randy Katz, Nets John Kubiatowicz, Arch James Landay, UI Jitendra Malik, Vision George Necula, PL Christos Papadimitriou, Theory David Patterson, Arch Kris Pister, Mems Larry Rowe, MM Alberto Sangiovanni- Vincentelli, CAD Doug Tygar, Security Robert Wilensky, DL/AI

11 Organization: The Expedition Cube Information Devices Information Utility Applications DesIgnDesIgn MethodologyMethodology MEMS Sensors/Actuators, Smart Dust, Radio Tags, Cameras, Displays, Communicators, PDAs Fluid Software, Cooperating Components, Diverse Device Support, Sensor-Centric Data Mgmt, Always Available, Tacit Information Exploitation (event modeling) Rapid Decision Making, Learning, Smart Spaces: Collaboration Rooms, Classrooms, Vehicles Base Program Option 1: Sys Arch for Diverse Devices Option 2: Oceanic Data Utility Option 4: Negotiation Arch for Cooperation Option 5: Tacit Knowledge Infrastructure Option 6: Classroom Testbed Option 7: Scalable Safe Component-Based Design Option 3: Capture and Re-Use

12 Base Program: Leader Katz Broad but necessarily shallow investigation into all technologies/applications of interest –Primary focus on Information Utility »No new HW design: commercially available information devices »Only small-scale testbed in Soda Hall –Fundamental enabling technologies for Fluid Software »Partitioning and management of state between soft and persistent state »Data and processing placement and movement »Component discovery and negotiation »Flexible capture, self-organization, info re-use –Limited Applications –Methodology: Formal Methods & User-Centered Design

13 System Architecture for Vastly Diverse Devices Leader Culler Design Issues for “Small Device OS” –Current: managing address spaces,thread scheduling, IP stack, windowing system, device drivers, file system, APIs, power management –How can OSs for tiny devices be made radically simpler, manageable, and automatically composable? Devices of Interest: Dust Motes

14 Communication-Centric Architecture Active Proxies –connected to the infrastructure –soft-state, bootstrap protocol –transcoding, Ubiquitous Devices –billions –sensors / actuators –PDAs / smartphones / PCs –heterogeneous Service Path Base Scalable Infrastructure –highly available –persistent state (safe) –databases, agents –service programming environment Service Paths –aggregate flows (rivers) –transcoding operators

15 Servers Clients Servers Infrastructure Services Open “The Large”: Service-Centric Platform Arch Enable distributed creation/deployment of scalable, available services – Service registry, aggregate execution env., transparency – Persistent distributed data structures – Massive fluid storage (“Oceanic” Storage) – Adaptive high-bandwidth flows (rivers) Build infrastructure via composition of services

16 “The Small”: Radically Simple OS for Management & Composition Basic Assumptions: –Communication is fundamental –Direct “user interface” is the exception not the norm –Critical resource is scheduling data movements, not arbitrary threads of computation Tiny OS: Little more than an FSM –Commands: event stream merged with sensor/actuator events –General thread compiled to sequence of bounded atomic xacts –Constant self-checking and telemetry –Rely on the infrastructure for complex processing Correctness-by-construction techniques for cooperating FSMs (tie in to HW/SW co-design)

17 Implementation & Deployment of Oceanic Data Info Utility Leader Kubiatowicz Ubiquitous devices require ubiquitous storage –Consumers of data move, change access devices, work in many different physical places, etc. Needed properties: –Strong Security: data must be encrypted whenever it is in the infrastructure –Coherence: too much data for naïve users to keep coherent “by hand” –Automatic replica management and optimization: huge quantities of data cannot be managed manually –Simple and automatic recovery from disasters: probability of failure increases with size of system –Utility model: world-scale system requires cooperation across administrative boundaries

18 Pac Bell Sprint IBM AT&T Canadian OceanStore IBM Utility-Based Infrastructure Confederations of (Mutually Suspicious) Utilities –Settlement system among service providers –Buy and sell capacity as needed

19 OceanStore Architecture/Technology Name and Data Location –Issue: Find nearby data without global communication –Approach: Data location is aform of gradient-search of local pools of data (use of attenuated Bloom-filters) High Availability and Disaster Recovery –Issue: Eliminate backup as independent/fallible technology –Approach: Erasure-codes/mobile replicas provide stable storage for archival copies and snapshots of live data Introspective Monitoring and Optimization –Issue: Optimize performance on a global scale –Approach: Monitoring and analysis of access/usage relationships Rapid Update in Untrusted Infrastructure –Issue: Updates should not reveal info to untrusted servers –Approach: Incremental cryptographic techniques/oblivious function techniques to perform update

20 Applications that Enhance Human Activity Tacit Information Mining: exploit info flows & relationships to improve collaborative work –3D “activity spaces” for representing decision-making activities, people, & information sources –Visual cues to denote strength of ties between agents, awareness levels, activity tracking, & attention span Smart Spaces –Electronic collaborative problem-based learning –Physical and Virtual Learning Spaces –Enabled by information appliances –UI design/exploitation of tacit information

21 Experimental Testbeds Network Infrastructure GSM BTS Millennium Cluster WLAN / Bluetooth Pager IBM WorkPad CF788 MC-16 Motorola Pagewriter 2000 Velo Adaptive Broadband LMDS H.323 GW Nino Smart Classrooms Audio/Video Capture Rooms Pervasive Computing Lab CoLab Soda Hall CalRen/Internet2/NGI Smart Dust LCD Displays Wearable Displays

22 Summary: Putting It All Together 1. eXtreme Devices 2. Data Utility 3. Capture/Reuse 4. Negotiation 5. Tacit Knowledge 6. Classroom 7. Design Methods 8. Scale-up Devices Utility Applications Fluid Software Info Extract/Re-use Decision Making Group Learning Component Discovery & Negotiation Self-Organization

23 Base Program Schedule Year 1Year 2Year 3 Eval. & Initial Design Tools Smart Space Testbed1st Gen Fluid R/T Environ. 1st Gen Comp Neg. Protocols 1st Gen Persistent Fluid Store 1st Gen Sensor-Centric Info Mgmt Design Document + Early Evaluation Cooperative Learning App Rapid Decision Making App Refined Doc + Experiments Refine & Use Perf Eval 2nd Gen Persistent Fluid Store 2nd Gen Sensor-Centric IM 2nd Gen Fluid R/T Environ. 2nd Gen Negotiation Final Doc + Experiments Refined Tools & Flow Design Methodology Information Utility Information Applications

24 Year 1 Milestones  Design/initial deployment “smart space” testbed; Initial usability evaluation/refinement;  Initial design, prototype, and early evaluation of fluid software run-time environ;  Initial design component advertisement protocols & i/f negotiation spec language;  Initial prototype/refinement of component advertisement protocols & interface negotiation specification language;  Initiate prototype & refinement of distributed, persistent storage system;  Initial design of sensor- centric/stream-capture oriented data mgmt system;  Initiate prototype & refinement of sensor-centric data mgmt system;  Design of distributed, persistent storage system;  Initial design of tool flow for infrastructure-embedded software functionality;  Initiate implementation of system design tools for early testing;  Completion of initial system architecture design document and early system evaluation;

25 Coherently managing billions of devices where none are “average” Information on demand, available wherever needed, on a global scale, in an untrusted infrastructure Pervasive management of massive stream- oriented information collection/inference in the wide-area Data movement & transformation; Paths, not threads; Persistent state/soft state partitioning; Non-blocking RMI for remote functionality; Support for MEMS devices, cameras, displays, etc. Serverless/homeless/freely flowing data; Opportunistic distribution, promiscuous caching, without administrative boundaries; High availability/disaster recovery, application- specific data consistency, security; Overlapping, partially consistent indices; Data freedom of movement; Expanding “search parties” to find data, using application-specific hints Extract, manage, analyze streams of sensor data; Path-based processing integrated with storage; Data reduction via filtering/aggregation; Distributed collection & processing; “Evidence accumulation” from inherently noisy sensors ProblemTechnical Approaches

26 ProblemTechnical Approaches Overwhelming config- uration complexity of large & heterogeneous systems Ineffectiveness of technology-mediated collaborative work; Better support for rapid decision making; Enabling Problem- based Learning in Enhanced Physical & Virtual Spaces; Correctness by Construction: Safe Component Design; Dynamic self-configuration: advertise provided services, discover components providing required services, negotiate interface contracts, monitor compliance, eliminate non-performing confederates; Infer communications flow, indirect relationships, availability, participation to enhance awareness & support opportunistic decision making; New collaborative applications: 3D “activity spaces” for representing decision-making activities, people, & info sources; Visual cues “weighting” relationships among agents, awareness levels, activity tracking & attention span Device/net-independent people-to-people comms via pervasive translation/adaptation; Information dissemination technologies; Wide-area information mgmt/access; Formal specifications and methods; Safety enforcement, design/development methods; Proof carrying code/secure protocol verification;

27 Summary and Conclusions 21st Century Computing –Making people’s exploitation of information more effective –Encompassing eXtreme diversity, distribution, and scale –Computing you can depend on Key Support Technologies –“Fluid software” computational paradigms –System and UI support for eXtreme devices –Pervasive, planetary-scale system utility functionality –Active, adaptive, safe and trusted components –New “power tool” applications that leverage community activity

28 Industrial Collaborators SRI