Using the NSA Presentation to NOAO Users Committee October 5, 2005

NSA is part of an End-to-End Data Flow System NOAO Science Archive –Release 2 running (since Jan, 2003) Includes all compliant reduced data from NOAO surveys OpenSky Node serves catalog data from NDWFS Contents available through NVO services (DataScope) Contains about 2 TB total –Release 3 in development Will support access to raw data from NOAO instruments Both proprietary and non-proprietary access Will include pipeline-reduced data as available Infrastructure for eventual full-feature archive Data Reduction Pipeline Development –CCD Mosaic Pipeline nearing completion –NEWFIRM Pipeline to be ready for instrument commissioning –Both based on in-house pipeline management system and IRAF modules

End-to-end System, Continued NOAO Virtual Observatory Portal (prototype) –Integrated user interface –Access to NSA and external data through VO protocols –Access to local and external tools and services –New visualization tools that allow much more sophisticated exploration of data Including time-domain Data Transport System –Captures data streams from all NOAO instruments –Safestore function replaces save-the-bits –Multiple copies at NOAO, NCSA –3 TB cached in last 12 months

How it works Although each element of program has value, the real breakthrough is the combination of all into end-to-end data flow and management system –New method for observers to acquire their data Most data will arrive already reduced –Automatic processing of data from heavily used (“survey-friendly”) instruments –Access to archived pointed observations and many small surveys Develop concept of a “virtual survey” –Unified interface/access to VO data and tools –Infrastructure to expand and develop new VO-enabled analysis tools Value Added –Multiplies value of all observations –Ultimate “broad impact” - science with minimal additional resources –Another way of providing access within system context –Appropriate role for national observatory: to guarantee long term access to and curation of data for community

Sky Time Query Access Analyze NSA+

Sky Time Query Access Analyze NSA+

Sky Time Query Access Analyze

Sky Time Query Access Analyze

Sky Time Query Access Analyze NSA+ SDSS+ Chandra+ XMM+

Sky Time Query Access Analyze

Sky Time Query Access Analyze Group by Spatial Overlap

Sky Time Query Access Analyze Group by Spatial Overlap 

Sky Time Query Access Analyze Group by Spatial Overlap

Sky Time Query Access Analyze

Sky Time Query Access Analyze Create Catalogs Re-bin Image View Create Cut-out View catalog Re-bin Image View Imcombine Create Cut-out Create Catalog R-bin Image View Imcombine Create Cut-out Image StatisticsImage LocationsData Volumes Image Locations

Milestones Had hoped to demo end-to-end system at January 2006 AAS Have had to scale back expectations due to staff losses –Demo portal only at AAS –End-to-end system begins to run in April-June timeframe –Ramp up of operations next summer Operations includes quality control, user support, maintaining throughput, fixing problems Eventually catch up and reduce cached data Next cycle of development will include (e.g.) –Portal transition from scientific prototype to fully-engineered system –Develop integrated catalog-image archive tools –Extend portal-archive to include virtual workspace, grid-enabled, user-configurable pipelines –Tools for community to prepare their datasets for inclusion

Other Issues Reviewing our previous decision to make all metadata available immediately –Probably will allow PI’s to request (with justification) proprietary status for metadata IRAF –Remains important toolkit, but, in its present form, not the future of data reduction –New context will be Web and VO –Portal will provide access to IRAF Web services –Continue to maintain, but minimize new development and change user support model Understand that many in community depend on IRAF Will not undermine their ability to continue using it