TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Data, Visualization and Scheduling (DVS) Update Kelly Gaither, DVS Area Director

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Data Movement  The alpha draft of the data toolkit has been completed and has been initially reviewed by Lee:  Needs some tweaking to make it complete and consistent with other toolkit definitions  Will be distributed to the data working group after that  Key items to focus on over the next several weeks:  Formal testing plan to ensure that deployed tools are tested and stay working (configuration and change management plan)  Complete analysis of data transfer speeds PSC Speedpage ( has the raw numbers for point to point transfer times. Group at PSC will be investigating and documenting bottlenecks, and what can be reasonably expected given the current infrastructure.

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Data Management  Phil Andrews is leading discussions about globally available file systems going forward. Current examples of this are GPFS-WAN and Lustre-WAN  Will be looking at viability of Amazon S3 storage for TG user community

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Data Workshop  January 9-11, San Diego  The outcome will be a draft of a TG wide data movement/data management plan going forward  If you are attending:  Please Mark Sheddon to register  Please make your hotel reservations by the end of this week

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Data Collections  Data Collections RAT Report is complete  Defined what constitutes a formally designated TG data collection:  Classes: Research data collection Resource or community data collection Reference data collection  Types: Type 1: Generally Accessible Data Collections –Satisfied basic requirements Type 2: Compute Associated Data Collections –Requires routine usage of TG compute or visualization resources to process data (e.g., Purdue Environmental Data Portal consolidates several earth observation data collections) Type 3: Globally Available Data Collections –Large data collections available on global file systems (e.g., NVO) Type 4: TG Affiliated Data Collections –Data collections demonstrating a link to TG, for example a demonstrated TG user community desiring access

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Data Collections  Data Collections Working Group:  Charter is complete  Working group is expected to begin meeting beginning of 2007  Going forward in 2007 we expect folks to approach TG for inclusion as a TG data collection:  Will be completing a process for becoming a TG data collection before the review

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Visualization  A half-day visualization tutorial “Remote/Collaborative TeraScale Visualization on the TeraGrid,” was taught at IEEE Visualization on October 29, 2006:  Taught by Kelly Gaither (TACC), Mike Papka (ANL), Joe Insley (ANL) and David Ebert (Purdue)  Visualization Workshop for Users:  Coordinating a full day workshop on the Monday of TG ’07 (June 4, 2007)  Limiting enrollment to maximum  Picking ~8 power users to begin working with now and visualize their data as examples for the workshop  Will be distributing a call for participation by Jan, 2007

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 TeraGrid Visualization Gateway  Q4FY06 Accomplishments  Presentations & demonstrations of TeraGrid Visualization beta portal at SC06  Collaborative and Remote Visualization Functionality Remote Paraview on UC/ANL systems Remote & Collaborative Visualization on Maverick  Offers support for TeraGrid User Portal accounts and community accounts Current TeraGrid User Portal users can login with their TGUP account and have full access to Viz Gateway. Community users can create their own accounts with restricted access.  Q1FY07 & Future Plans  Continue development to production quality portal  Integrate Additional Visualization Tools Expand current set of visualization tools and work with other RP sites to see what they can offer (e.g., Purdue has expressed interest in including a Maya portlet). Look into registering ‘visualization services’ with the portal, either through web services or importing additional functionality.  Milestones: Have portal v1.0 complete and in production by TG07 Conference (and TG Viz User’s Workshop) Paper submission to TG 07 Conference

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Remote Visualization Portlet

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 ParaView Portlet

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Scheduling  Co-scheduling/metascheduling RAT:  Leadership and Members Warren Smith (TACC) and Patricia Kovatch (SDSC) Members from each RP site  Accomplishments to Date:  Performed a user survey, summarized and discussed the results  Performed an RP survey, summarized and discussed the results  Performed a review of metascheduling tools, summarized and discussed the results  Drafted a set of recommendations  Finishing final report (1/5/2007)  Developing a scheduling working group charter

TeraGrid Quarterly Meeting Dec 5 - 7, 2006 Scheduling  RAT Recommendation Topics:  Advance Reservations  Co-scheduling  On-demand Scheduling Highest priority Preemption  Automatic Resource Selection  Ensemble  Workflow  Four Primary Tasks:  Update user metascheduling requirements  Compile information about scheduling environments on RP systems as well as RP requirements and preferences in the area of metascheduling  Identify and perform a paper evaluation of metascheduling tools  Develop recommendations for metascheduling in TG