Science Archive for Sky Surveys Data Providers and the VO - NeSC 2003 March Wide Field Astronomy Unit Institute for Astronomy.

Slides:

Advertisements

Similar presentations

Trying to Use Databases for Science Jim Gray Microsoft Research

Advertisements

Australian Virtual Observatory A distributed volume rendering grid service Gridbus 2003 June 7 Melbourne University David Barnes School of Physics, The.

Publishing catalogues: HIPASS and SUMSS Australian Virtual Observatory 2003 Workshop 17th-18th November 2003 Sydney David Barnes, Brett Beeson, Travis.

The Australian Virtual Observatory e-Science Meeting School of Physics, March 2003 David Barnes.

The Storage Resource Broker and.

The Storage Resource Broker and.

NGAS – The Next Generation Archive System Jens Knudstrup NGAS The Next Generation Archive System.

Software for Science Support Systems EVLA Advisory Committee Meeting, March 19-20, 2009 David M. Harland & Bryan Butler.

Solar and STP Physics with AstroGrid 1. Mullard Space Science Laboratory, University College London. 2. School of Physics and Astronomy, University of.

VISTA/WFCAM pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

Leicester Database & Archive Service J. D. Law-Green, J. P. Osborne, R. S. Warwick X-Ray & Observational Astronomy Group, University of Leicester What.

Sara SartoliAkbar Siami Namin NSF-SFS workshop July 14-18, 2014.

Leicester Database & Archive Service J. D. Law-Green, S. W. Poulton, J. Osborne, R. S. Warwick Dept. of Physics & Astronomy, University of Leicester LEDAS.

SDSS Web Services Tamás Budavári Johns Hopkins University Coding against the Universe.

Computer Science & Engineering 2111 CSE 2111 Lecture Querying a Database 1CSE 2111 Lecture- Querying a Database.

Data provenance in astronomy Bob Mann Wide-Field Astronomy Unit University of Edinburgh

Aus-VO: Progress in the Australian Virtual Observatory Tara Murphy Australia Telescope National Facility.

Hubble Legacy Archive Lee Quick - TIPS meeting July 19, 2012 Goals Data History Current Work Demo.

Discussion and conclusion The OGC SOS describes a global standard for storing and recalling sensor data and the associated metadata. The standard covers.

VO as a Data Grid, NeSC ‘03 WFCAM Science Archive Nigel Hambly Wide Field Astronomy Unit Institute for Astronomy, University of Edinburgh.

Introduction to Sky Survey Problems Bob Mann. Introduction to sky survey database problems Astronomical data Astronomical databases –The Virtual Observatory.

Cool white dwarfs in the Sloan & SuperCOSMOS Sky Surveys Nigel Hambly, Wide Field Astronomy Unit, IfA, University of Edinburgh.

INFN-Pisa Glast Database in Pisa A practical solution based on MSAccess Luca Latronico INFN Pisa.

Oracle Application Express (Oracle APEX), formerly called HTML DB, is a Free rapid web application development tool for the Oracle database.

F. Genova, VOTECH kickoff meeting, 2004/11/ Interoperability standards F. Genova, M. Allen, T. Boch, F. Bonnarel, S. Derriere, P. Fernique, F. Ochsenbein,

EdSkyQuery-G Overview Brian Hills, December

Functions and Demo of Astrogrid 1.1 China-VO Haijun Tian.

The ASDC SED Builder Milvia Capalbi (INAF-ASDC) in collaboration with Paolo Giommi (ASI-ASDC), Giulia Stratta (INAF-ASDC), Roberto Primavera (ElsagDatamat)

Talk structure who are we ? what is a VO ? what are the challenges ? what is an e-project ? Andy Lawrence Garching June 2002.

Astronomical data curation and the Wide-Field Astronomy Unit Bob Mann Wide-Field Astronomy Unit Institute for Astronomy School of Physics University of.

Science with the Virtual Observatory Brian R. Kent NRAO.

1 Foundations V: Infrastructure and Architecture, Middleware Deborah McGuinness TA Weijing Chen Semantic eScience Week 10, November 7, 2011.

CS480 Computer Science Seminar Introduction to Microsoft Solutions Framework (MSF)

NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 1 Virtual Observatories...or how to do your research from a beach in the Bahamas rather.

Prototype system of the Japanese Virtual Observatory The Japanese Virtual Observatory (JVO) aims at providing easy access to federated astronomical databases.

JVO JVO Portal Japanese Virtual Observatory (JVO) Prototype 2 Masahiro Tanaka, Yuji Shirasaki, Satoshi Honda, Yoshihiko Mizumoto, Masatoshi Ohishi (NAOJ),

ISYS 562 Microcomputer Business Applications David Chao.

WFCAM Science Archive Critical Design Review, April 2003 The SuperCOSMOS Science Archive (SSA) WFCAM Science Archive prototype Existing ad hoc flat file.

Strasbourg astronomical Data Centre (DS) Françoise GENOVA.

Canadian Virtual Observatory David Schade Canadian Astronomy Data Centre Pat Dowler, Daniel Durand, Luc Simard, Norm Hill, Severin Gaudet.

AstroWise((G)A)VO Meeting 6 May 2004 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise.

AstroGrid: The UK’s Virtual Observatory Dr Dugan Witherick – Astrophysics Group, UCL Wednesday 5 th December 2007 The University of Warwick.

1 10-June-2004Andy Lawrence : PPARC data curation panel meeting AstroGrid, Data Centres, & Edinburgh What is curation ? Data Centres in the VO era Data.

Understand how the future will work exchange information on key projects understand PPARC priorities debate (conclude?) community approach to the PPARC.

EScience May 2007 From Photons to Petabytes: Astronomy in the Era of Large Scale Surveys and Virtual Observatories R. Chris Smith NOAO/CTIO, LSST.

The Project The Virtual Observatory Technical Progress Andy Lawrence Nottingham All-Hands meeting Sept 2003 AstroGrid

Astronomical Data Archiving and Curation Clive Page AstroGrid Project University of Leicester 2004 March 22.

Virtual Survey System sept 04 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise a Virtual.

Solar and space physics datasets within a Virtual Observatory: the AstroGrid experience Silvia Dalla * and Nicholas A Walton  * School of Physics & Astronomy,

● Radio telescope arrays – km diameter – Resolution arcmin to micro-arcsec at radio wavelengths ● Similar (baseline/ wavelength) for other regimes.

March 1st, 2006Prospective PNG PNG: Databases - Virtual Observatory.

The International Virtual Observatory Alliance (IVOA) interoperability in action.

Catalog Data Interface Tool Virtual Observatory-India (VO-I) May 16-20, 2005 Catalog Data Interface.

German Astrophysical Virtual Observatory Overview and Results So Far W. Voges, G. Lemson, H.-M. Adorf.

Virtual Observatories, Press Release Images, and Web Services Dr. Frank Summers Space Telescope Science Institute November 3, 2005.

AstroGrid NAM 2001 Andy Lawrence Cambridge NAM 2001 Andy Lawrence Cambridge Belfast Cambridge Edinburgh Jodrell Leicester MSSL.

Pan-STARRS PS1 Published Science Products Subsystem Presentation to the PS1 Science Council August 1, 2007.

12 Oct 2003VO Tutorial, ADASS Strasbourg, Data Access Layer (DAL) Tutorial Doug Tody, National Radio Astronomy Observatory T HE US N ATIONAL V IRTUAL.

Web based spectrum databases and utilities László Dobos Tamás Budavári István Csabai MAGPOP kick-off meeting, January Cassis.

Biomedical Informatics Research Network The Storage Resource Broker & Integration with NMI Middleware Arcot Rajasekar, BIRN-CC SDSC October 9th 2002 BIRN.

Introduction to the VO ESAVO ESA/ESAC – Madrid, Spain.

ViaSQL Technical Overview. Viaserv, Inc. 2 ViaSQL Support for S/390 n Originally a VSE product n OS/390 version released in 1999 n Identical features.

The ATLAS DAQ System Online Configurations Database Service Challenge J. Almeida, M. Dobson, A. Kazarov, G. Lehmann-Miotto, J.E. Sloper, I. Soloviev and.

Publishing Combined Image & Spectral Data Packages Introduction to MEx M. Sierra, J.-C. Malapert, B. Rino VO ESO - Garching Virtual Observatory Info-Workshop.

Evanthia Hatziminaoglou, ESO - Garching Virtual Observatory Info-Workshop, SOFIA January 2008 VO Tools Overview.

1 Workflows Access and Massage VO Data José Enrique Ruiz on behalf of the Wf4Ever Team IVOA INTEROP SPRING MEETING 2013 HEIDELBERG, MAY16th 2013.

AstroGrid and Virtual Observatories for Radio Interferometry arrays/ proposals Anita Richards Paul Harrison Noel Winstanley (Jodrell Bank Centre for Astrophysics,

Critical Design Review, April 2003

aspects of archive system design

Presentation transcript:

Science Archive for Sky Surveys Data Providers and the VO - NeSC 2003 March Wide Field Astronomy Unit Institute for Astronomy

Data Sources UKIRT Wide-Field Camera (WFCAM) early 2004 –100 GB/night (four 2048 x 2048 infrared arrays) –~24 TB/year pixel data + ~2 TB/year object data from data-reduction pipeline operated by CASU VISTA Infrared camera from GB/night (sixteen 2048 x 2048 infrared arrays) ?? from a future optical camera...

What is the Science Archive? Not a simple repository of data, but a system that: enables access to large data volumes provides reduced-size results sets provides tools to enable science exploitation can become part of a Virtual Observatory

Design approach Requirements & specification developed in collaboration with stakeholders (UKIDSS &c) Design governed by agreed requirements Build on existing experience (e.g. SSS, 6dF GS) Scalability

Phased approach V1 Archive System for WFCAM first light Jan 2004 V2 one year after survey operations begin allowing further studies of DBMS scalability V3 Grid-enabled, at some future point

Overall project plan VISTA first light

Version 1 Archive System Following SDSS DR-1 (not EDR) for catalogues: “SkyServer” system employing MS SQL Server running on W2K PC(s); browser-based GUIs with clickable options and SQL editor Additional simple web-page interfaces for pixel data Facilities for joins with other catalogues e.g. 2MASS

SSS pixel data interface WWW interface to extract pixel or object data by (RA,Dec) selection Or, access via CDS Aladin tool

SSS pixel/obj data interface After extracting data from WWW interface, upload them straight into GAIA.

Form access to 6dFGS generates SQL for you

e.g. choose table from menu

and get the results

or write SQL directly

Migrate SSS to SQL Server DB Schemas for SSS developed subset of SSS installed on a laptop Put SSS catalogue data in SQL Server to gain experience and test scalability in preparation for Version 1 of Archive

SSS SQLServer database ultimate SSS archive and precursor to WFCAM/VISTA

Version 2.0 and beyond Version 2.0 will, in addition to V1.0: - enhanced GUI functionality (e.g. plotting) - on-the-fly cats and pix over arbitrary areas - generate stacked frames with user-specified options - provide new source extraction options Version 3.0 will, in addition to V2.0: - return data in VOTable using VO transfer protocols - recast the online services as web services - develop (or install) advanced visualisation tools - develop (or install) advanced analysis tools - allow upload of user analysis codes - ……

Remote client GUI access SDSS JavaQT Complex queries using SQL