University of Illinois at Urbana-ChampaignNational Center for Supercomputing Applications Towards Truly Ubiquitous Cyberinfrastructure LAGrid 07 Jim Myers Associate Director for Cyberenvironments and Technologies, National Center for Supercomputing Applications (NCSA), University of Illinois at Urbana-Champaign

National Center for Supercomputing Applications Cyber-resources Innovative Systems Communities and Applications Cyberenvironments

Outline Whats Changing in Science? What Role should Cyberinfrastructure (CI) play? What Do Ubiquitous (and Persistent) mean for CI Development? Designing for Ubiquity Some Examples Conclusions National Center for Supercomputing Applications

How is Science Changing? Quantitative Modeling and Simulation Better Data (e.g. Higher Signal to Noise) More Data (e.g. High Throughput) –Closer ties between research and application –Investigation of subtle, non-linear, multi-dimensional phenomena –Statistical analysis of complex systems

National Center for Supercomputing Applications The Research Process Its just the Scientific Method…

National Center for Supercomputing Applications The Research Process F g ~m Conceptual Logical Physical Assumptions Reference Data Controls… Reduction Statistics Analysis of Alternatives… With Experimental Design…

National Center for Supercomputing Applications The Research Process Scientific Instrument Method F g ~m High-speed camera And Multiple, Coupled Objectives…

National Center for Supercomputing Applications The Research Process Collaboration Reference Data Curation Model Validation Sub-discipline Creation Best-practice Dissemination Application Education … Scientific Instrument Method And Community Processes …

National Center for Supercomputing Applications The Research Process Non-linear, high- dimensional, coupled, multi-scale phenomena Scientific Instrument Method And Its No Longer F g ~m …

National Center for Supercomputing Applications Amdahls Law for Scientific Progress: Data discovery Translation Experiment setup Group coordination Tool integration Training Feature Extraction Data interpretation Acceptance of new models/tools Dissemination of best practices Interdisciplinary communication Data productionProcessing power Data transfer/storage !

National Center for Supercomputing Applications Whats Needed to Support the Research Lifecycle? Discover Mine Translate Reference Extract Experiment Design Annotation Provenance Gap Analysis Reference Data Publish Share Coordinate Curate Validate Relate 1 2 Valid Range Project Execution Engineering Views Standards / Best practice Sensor Data Algorithms/ Services

National Center for Supercomputing Applications There is a class of bovine-related problems for which shape is not important Yet shape is clearly needed in a general cow model Should we reach consensus here? Is there one best way to map volume to height? Consider a Spherical Cow… Moo! ACME Trucking

National Center for Supercomputing Applications Key Issues for Ubiquitous & Persistent CI CI must be built before the parts are done It must be evolvable by independent parties It must enable coordination without central control It must allow science to evolve / progress –No fixed domain model Researchers/educators must be able to work in multiple communities/value chains (across CI projects) It must convey knowledge as well as tools to end users It must align the interests of CI funders, developers, providers, users, …

Can this be done? National Center for Supercomputing Applications

Yes! Design Principles for loosely coupled, scalable (not scaled) systems and organizations Agile, community/science driven development processes over longer-term community/science driven design …e-Science, Semantic Grid, Web 2.0 … …intelligence at the edges…

National Center for Supercomputing Applications Key Cyberenvironment Design Concepts Explicit Representations Separating How from What: –Content (metadata, global IDs, …) –Process (workflow, provenance, …) –Virtual Organizations (policies, resources, semantics, translation) –GUI Integration (portals, rich clients, …) –…

University of Illinois at Urbana-ChampaignNational Center for Supercomputing Applications MAEViz – an Example Cyberenvironment ( Consequence-Based Risk Management for Seismic Events) Mid-America Earthquake Center Engineering View of MAE Center Research Portal-based Collaboration Environment Distributed Data/metadata Sources Multi-disciplinary Collaboration Hazard Definition Inventory Selection Fragility Models Damage Prediction Decision Support

National Center for Supercomputing Applications Content Management Whatever thing we are talking about, we want –To know its type, –Have descriptive information so we can find and categorize it, –Be able to version it, –Specify who owns and can access it, –Define its relationships to other things, –Manage copies of it / know when you have it, –Be able to translate it, –Dynamically add new information we learn about it, –…

National Center for Supercomputing Applications Content Aware ARKs, DOI, LSID WebDAV, JCR, RDF, SAM, Tupelo Desktop Secure Enterprise Data Public Reference Data Data/Metadata

National Center for Supercomputing Applications Process Management Framework Workflow description as a means of communicating experiment protocol –Actors built as modules, web services, grid jobs… –Process execution managed through direct calls, service calls, data transfer, events, manual processes, … Workflow generated by applications, by example, graphically, or discovered from provenance Execution performed using an engine with appropriate speed, reliability, availability of modules, etc. Workflow templates and provenance records treated as sharable content (versioned, compared, documented, …) Process descriptions captured at multiple levels of detail (scientific, mathematical, engineering, debugging, …) Community Provenance and Process extend across workflows

National Center for Supercomputing Applications Process Management Workflow Creation Hierarchical Workflow Application Interface Provenance Workflow-by-Example X=f(y) Y = f2(z) Scripting

National Center for Supercomputing Applications Process Aware Workflow, Provenance, RDF Discover Process Capture Execute Report

National Center for Supercomputing Applications Virtual Organizations Grid/portal concept for managing –Single sign-on security –access control policies –toolsets and views –data sources –processes and results –resource pools –vocabularies and models –… Tools query VO manager to configure themselves based on VO context/policies/preferences

National Center for Supercomputing Applications Pluggable User Interfaces Portlet/Rich-Client concept, broadened to include VO configuration of –Content sources –Events –Workflow/Provenance repositories –Data models/ontologies –Translations Portal technologies: JSR 168, Teamlets, WSRP, JSR 286, … Rich Clients: Eclipse/OSGi, JSR 170, 283, …

National Center for Supercomputing Applications Group Aware Collaboratory, Portal, … Plan, Coordinate, Share, Compare Wiki Task List Chat Document Repository Scenario Repository Training Materials SSO

National Center for Supercomputing Applications Dynamic Plug-ins, WSRP, Provenance Eclipse RCP WorkflowDataGIS MAEviz Plug-in Framework Auto-update New Third-Party Analyses Compare, Contrast, Validate

National Center for Supercomputing Applications Rich, VO-oriented plug-in mechanism Third-party Plug-in Adds to menu Adds to interface Adds to workflow Adds to provenance Joins Security Context Maps data model X X X X

Environmental Observatories Rely on advances in: sensors and sensor networks at intensively instrumented sites shared by the research community cyberinfrastructure with high bandwidth to connect the sites, data repositories, and researchers into collaboratories distributed modeling platforms From USGS

National Center for Supercomputing Applications Observatories as a Community Focus

National Center for Supercomputing Applications Sensors Data Products Derived Data Products Storage QA/QC Archive Operations/Expt. Design Cache Knowledge Store Community Provisioning Community Coordination/ Knowledge Creation Events Model Dev/ Validation Research & Education Projects Observatory Operation and Evolution On-demand Services and HPC Third-party Resources Data Access Environmental Observatory Processes Documentation Coordination Recommendations

National Center for Supercomputing Applications Ubiquity = Supporting Scientific Discourse Cyberenvironments represent rethinking current practice to create CI –That is enabling rather than stifling –That evolves as fast a research evolves –That connects research and practice –That empowers individuals to contribute new resources –That can be ubiquitous and persistent –That enables resource repurposing to address new questions –That opens new career paths for CI developers, data scientists, systems engineers, …

National Center for Supercomputing Applications Cyberinfrastructure Challenges How can CI increase the productivity and competitiveness of the scientific community? How can CI developers enhance their capacity to respond to user needs more rapidly and more effectively? How should CI technical design and organizational structures change to enable solutions at scale – as a ubiquitous, persistent infrastructure for science and engineering research and education?

National Center for Supercomputing Applications Cyberenvironments Mosaic and Cyberenvironments Mosaic –By early 1990s, the internet had a wealth of resources, but they were inaccessible to most scientists –Individual publishing –Browsing versus retrieving –See Web The Machine is Us/ing Us Cyberenvironments –By the early 2000s, the internet and grid had a wealth of interactive resources, but they were inaccessible to most scientists –Individual information models –Fusion versus gathering

National Center for Supercomputing Applications Acknowledgments NCSA CET Staff NCSA Collaborators CI Community National Science Foundation/State of Illinois/ONR Mathematical, Information and Computational Sciences Division of the Office of Science … and Thank You