GEC3 Oct www.geni.net1 Experiment Workflow and Services Working Group System Engineering Report Vicraj (Vic) Thomas, Ph.D. Working Group System Engineer groups.geni.net GENI working group wiki

GEC3 Oct We Are Here… The Experiment Services and Workflow WG focuses on experimenter- users needs for planning, scheduling, running, debugging, analyzing and archiving experiments.

GEC3 Oct Lifecycle of a GENI Experiment User view of an experiment Tools and services supporting this lifecycle map to different subareas identified by Jeff Chase –Slice control Construct, deploy, monitor and adapt slices –Information plane Instrumentation: Specification, collection, processing –Experiment building blocks Code artifacts: Specification, qualification, sharing/reuse, deployment, archival, etc. Objective: Think about how your ideas and projects relate to workflow presented here –How they support this workflow –Identify areas of disagreement

GEC3 Oct Lifecycle of a GENI Experiment: Top- Level View Experiment Design S/w and H/w Development (Lab) Experiment Planning Experiment Execution Experiment Sunsetting Experiment Deployment GENI Tools and Services support these phases of the lifecycle Outside the scope of GENI User Registration

GEC3 Oct GENI Relevant Lifecycle Phases Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install and Configure Software Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

GEC3 Oct User Registration Needed: Admin tools to register users and grant them appropriate authorizations –Administrator uses real- world documents to authenticate user –User is registered with a Principal Registry/Identity Provider –User is granted appropriate authorizations Browse resources Lease resources and run experiments Admin and Account Tools

GEC3 Oct Experiment Planning Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install and Configure Software Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

GEC3 Oct Establishing Credentials User establishes identity with Clearinghouse –Assumption: Already registered with an Identity Provider/Principal Registry

GEC3 Oct Resource Discovery Multiple ways of finding resources –Automatically from an experiment specification Needed: A tool that can interpret an experiment specification and query appropriate clearinghouses for resources –Programmatically Needed: An API on Clearinghouses to query for resources –Browse and select Needed: A graphical tool for browsing resources –May lease resources during discovery (for immediate or future use) Researcher credentials must allow leasing Assumption: All components and aggregates are registered with one or more Clearinghouses

GEC3 Oct Tool Discovery Determine tools needed to program GENI components discovered by researcher –Need: Component Rspec to provide links to information on using/programming component –Need: Component Rspec to provide links to information on resource and other constraints imposed by component

GEC3 Oct Develop GENI Version Version of experiment that will run on GENI components –Developed and tested in the lab Based on component information obtained during resource and tool discovery Develop instrumentation plan and implement GENI Tools/Services: None needed

GEC3 Oct GENI Relevant Lifecycle Phases Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install and Configure Software Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

GEC3 Oct Establish Deployment Credentials Assumption: Researcher has rights to lease resources –Credentials to lease resources and use them may be different from those for browsing resources Needed: Researcher’s Principal Registry/Identity Provider to be trusted by Clearinghouses

GEC3 Oct Lease Resources Similar to browsing for resources –Automatically from an experiment specification Needed: Tool to interpret experiment specification and lease resources at appropriate Clearinghouses –Programmatically Needed: API on Clearinghouses to lease resources –Browse and lease Needed: Graphical tool to lease resources Researcher specifies duration of lease and start time Leasing Interface Details of Control Plane operations (slice management, tickets, etc) are hidden from the casual user

GEC3 Oct Install and Configure Software Download software into components and install them –Experimenter must have valid lease for resource Configure components such as firewalls and Internet gateways Needed: Tool to manage downloads, track configurations, etc. Needed: Mechanisms for composing experiments Experimental hardware: Specific to type of hardware and where it goes –Post Spiral 1

GEC3 Oct Validate Deployment Needed: Tools to ensure slice is properly configured –Required resources have been obtained –Components connected and communicating

GEC3 Oct GENI Relevant Lifecycle Phases Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install Hardware/Soft ware Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

GEC3 Oct Experiment Control Needed: Services and tools to support –Start/pause/resume/stop experiments Granularity: Slice? Sliver? Experiment relevant groupings? All of the above? –Grow/shrink slice: Add/remove resources –Control flow of traffic from/to Internet –Connect to other experiments (experiment composition) –Distributed synchronization for multi-phased experiments –Debugging and checkpointing Resources being used by experiment Experiment state –Failure management Handling failures of components and links:

GEC3 Oct Data Collection and Analysis Needed: Non-intrusive instrumentation –Data collection and storage Needed: Tools for monitoring and visualizing resources, flows, etc. in real-time while experiment is running

GEC3 Oct GENI Relevant Lifecycle Phases Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install Hardware/Soft ware Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

GEC3 Oct Experiment Archival Make experiment available to the research community to replicate, extend or incorporate into another experiment Needed: Checklist of tasks to be completed before experiment is archived –Documentation of experiment objectives, code, resources used; keywords; experiment description in a formal specification language; published reports; etc. Needed: Tools to search archives Needed: Embedding experiment specifications

GEC3 Oct Transition Technology to Product Outside scope of GENI tools and services?

GEC3 Oct Working Group Deliverables and GPO Support

GEC3 Oct WG Deliverables GENI Experiment Lifecycle Spiral 1 Capabilities-based Use Case –Control plane oriented Experiment Support Subsystem Technical Description Experiment Support Tools and Services Technology Survey

GEC3 Oct Goals Understand the GPO system engineering role and contributions to this working group Track hard problems and open issues relevant to this working group – And identify researchers and projects Identify how each of us can contribute to addressing these problems – Including developing cogent descriptions of these problems

GEC3 Oct Who am I? GENI Systems Engineer for Distributed Systems and Security Background in highly dependable distributed systems and networks –Distributed control system infrastructures for process control systems –Distributed collaboration system for operators of process control plants –PI on DARPA NEST and Ultra*Log projects –Lead on DARPA proposal that led to the development of a very successful intrusion report aggregator –Ground-breaking work on wireless networks for on- aircraft flight-essential applications –Industrial Technical Area Lead for Sensor Information Processing and Delivery on US-UK program (ITA program) on Network Sciences

GEC3 Oct My Role in this Working Group Frame issues from top-down through document development –Collect & frame issues –Collect input from group & revise Synthesize input from bottom-up –Test for & document consensus (or lack of it) –Take & distribute notes –Maintain wiki Track hard problems and open issues – Solicit help with developing cogent descriptions of problems – Identify researchers and projects that can address these problems

GEC3 Oct GPO System Engineering and Working Groups SEs draft documents Internal GPO review Post publically –On wiki initially, repository up shortly Discuss doc on working group list –Possible one-on-one followups –Please send text Revision is posted –Repeat process

GEC3 Oct GEC3 GENI Spiral 1 Integration: 5 Control Framework Clusters 1578 Overlay Hosting Nodes 1579 ProtoGENI 1595 GpENI 1595 GpENI 1646 CMU Testbeds 1609 DETER Trial Integration 1609 DETER Trial Integration 1642 Instrumentation Tools 1613 Enterprise GENI 1600 PlanetLab 1658 Mid-Atlantic Crossroads 1658 Mid-Atlantic Crossroads 1602 Sensor/Actuat or Network 1582 ORCA/BEN 1599 Vehicular Mobile Network 1660 ORBIT Framework 1601 Virtual Tunnels 1657 WIMAX 1657 WIMAX 1663 Digital Object Registry 1663 Digital Object Registry 1633 Kansei Sensor Network 1628 Measurement System 1604 GENI Meta Operations 1643 Programmable Edge Node 1621 GUSH Tools 1622 Provisioning Service 1650 Regional Opt-In 1650 Regional Opt-In 1632 Security Architecture 1645 Million- Node GENI 1610 GENI at 4-Year Colleges 1610 GENI at 4-Year Colleges 1653 Data Plane Measurements Cluster A Cluster ECluster D Cluster CCluster B Key: Projects with active Spiral 1 clearinghouse interfaces STUDY ALL PICK ONE 1631 Embedded Real-time Measurements 1631 Embedded Real-time Measurements 1619 Optical Access Networks 1619 Optical Access Networks Column labels show common control framework Highlighted projects are relevant to this wg. Is this the right set?

GEC3 Oct Follow-up Notes, slides, actions, etc will be sent to the working group mail list and posted on the wiki page: Please give Jeff/me copies of your slides immediately after this session Please sign Non-Exclusive Copyright License