The Astro-Wise Pipeline Conceptual Design Review Roeland Rengelink Requirements Development Framework Prototype.

Slides:

Advertisements

Similar presentations

Object Oriented Analysis And Design-IT0207 iiI Semester

Advertisements

© 2004 Visible Systems Corporation. All rights reserved. 1 (800) 6VISIBLE Holistic View of the Enterprise Business Development Operations.

Building Enterprise Applications Using Visual Studio ®.NET Enterprise Architect.

ARCS Data Analysis Software An overview of the ARCS software management plan Michael Aivazis California Institute of Technology ARCS Baseline Review March.

Software Frameworks for Acquisition and Control European PhD – 2009 Horácio Fernandes.

McGraw-Hill/Irwin Copyright © 2007 by The McGraw-Hill Companies, Inc. All rights reserved. Chapter 1 Introduction to Database Management.

1 IS 4420 Database Fundamentals Chapter 2: Database Development Process Leon Chen.

© , Michael Aivazis DANSE Software Issues Michael Aivazis California Institute of Technology DANSE Software Workshop September 3-8, 2003.

VO-lecture Valentijn 17/1/2007 Astro-Wise Wide Field imaging Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving,

The ARCS Data Analysis Software Michael Aivazis California Institute of Technology.

Supplement 02CASE Tools1 Supplement 02 - Case Tools And Franchise Colleges By MANSHA NAWAZ.

DATABASE DEVELOPMENT STRATEGIES TOP DOWNTOP DOWN –Large scale application driven by strategic objectives –General  Specific –Organization-wide (“data.

Building software from reusable components.

Visual Basic 6.0 Derived from BASIC Developed by Microsoft in 1998 An event driven programming language Associated with a development environment.

Database Administration Chapter 16. Need for Databases  Data is used by different people, in different departments, for different reasons  Interpretation.

Chapter 1 Introduction to Database Management. McGraw-Hill/Irwin © 2004 The McGraw-Hill Companies, Inc. All rights reserved. Welcome! Database technology:

MAHI Research Database Project Status Report August 9, 2001.

October 30, 2008 Extensible Workflow Management for Simmod ESUG32, Frankfurt, Oct 30, 2008 Alexander Scharnweber (DLR) October 30, 2008 Slide 1 > Extensible.

Performance and Exception Monitoring Project Tim Smith CERN/IT.

Astro-Wise workshop Nov 2005 Astro-Wise workshop Lorentz center Nov 2005 Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden, Groningen Lorentz.

The Astro-wise pipeline Python Data processing - the low-level interfaces Data administration - persistent objects.

Introduction to Database Management. 1-2 Outline  Database characteristics  DBMS features  Architectures  Organizational roles.

CS480 Computer Science Seminar Introduction to Microsoft Solutions Framework (MSF)

Chapter 11: Software Prototyping Omar Meqdadi SE 273 Lecture 11 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.

Oracle9i Performance Tuning Chapter 1 Performance Tuning Overview.

Resource Management Working Group SSS Quarterly Meeting November 28, 2001 Dallas, Tx.

Chapter 1 : Introduction §Purpose of Database Systems §View of Data §Data Models §Data Definition Language §Data Manipulation Language §Transaction Management.

Information System Development Courses Figure: ISD Course Structure.

Chapter 2: SQL – The Basics Objectives: 1.The SQL execution environment 2.SELECT statement 3.SQL Developer & SQL*Plus.

1 st -4 th December st BioXHIT Annual Meeting WorkPackage 5.2: Implementation of Data management and Project Tracking in Structure Solution Peter.

Accomplishments: Implementation Login form Dictionary Inputs Calculations(Partially Completed) Output(Partially Completed) Made a test database.

10 June 2002Towards an International VO - Garching bei Munchen 1 ASTRO-WISE An Astronomical Wide-field Imaging System for Europe Konrad Kuijken, Edwin.

Software Development Cycle What is Software? Instructions (computer programs) that when executed provide desired function and performance Data structures.

Oracle Data Integrator Procedures, Advanced Workflows.

AstroWise((G)A)VO Meeting 6 May 2004 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Integration of Astro-WISE with Grid storage.

EURO-VO workshop 1 July 2005 A Virtual Survey SYSTEM Astro-Wise NOVA/Kapteyn – OA Capodimonte ESO – Terapix – US Munich/MPE National WFI datacenters NL-I-D-Fr/ESO.

Project Methodology May 2, System Development Life Cycle Overview.

Virtual Survey System sept 04 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise a Virtual.

17 th October 2005CCP4 Database Meeting (York) CCP4(i)/BIOXHIT Database Project: Scope, Aims, Plans, Status and all that jazz Peter Briggs, Wanjuan Yang.

The Volcano Optimizer Generator Extensibility and Efficient Search.

Portable Infrastructure for the Metafor Metadata System Charlotte Pascoe 1, Gerry Devine 2 1 NCAS-BADC, 2 NCAS-CMS University of Reading PIMMS provides.

J. Scott Hawker p. 1Some material © Rational Corp. Rational Unified Process Overview See and use the RUP Browser on lab machines.

Database Administration

Software Prototyping Rapid software development to validate requirements.

SimDB Implementation & Browser IVOA InterOp 2008 Meeting, Theory Session 1. Baltimore, 26/10/2008 Laurent Bourgès This work makes use of EURO-VO software,

Architecture View Models A model is a complete, simplified description of a system from a particular perspective or viewpoint. There is no single view.

Mantid Stakeholder Review Nick Draper 01/11/2007.

04 - OOD Intro.CSC4071 Software Design ‘Requirements’ defines –The goals the system needs to satisfy. ‘Specification’ defines –The externally-observable.

CSIS 4850: CS Senior Project – Spring 2009 CSIS 4850: Senior Project Spring 2009 Object-Oriented Design.

Framework for Collecting, Reporting, & Sharing Senior Design III - Spring 2007 Louis Von Eye.

The ECOST Web-based platform for data providers and for data users.

Building Enterprise Applications Using Visual Studio®

Introduction to DBMS Purpose of Database Systems View of Data

Databases and DBMSs Todd S. Bacastow January 2005.

Database Development Lifecycle

Prototyping in the software process

Software Project Configuration Management

Software Prototyping.

An Overview of Requirements Engineering Tools and Methodologies*

EIN 6133 Enterprise Engineering

Chapter 2: Database System Concepts and Architecture

Data, Databases, and DBMSs

OBJECT-ORIENTED APPROACH TO OFFICIAL STATISTICS

Software Design Lecture : 15.

Introduction to DBMS Purpose of Database Systems View of Data

The User Interface Design Process

Overview Activities from additional UP disciplines are needed to bring a system into being Implementation Testing Deployment Configuration and change management.

Graduation Project Project Name

Our Process CMSC 345, Version 1/04.

Presentation transcript:

The Astro-Wise Pipeline Conceptual Design Review Roeland Rengelink Requirements Development Framework Prototype

Baseline Requirements Data Flow System User Requirements (URD) overview of the scientific requirements, includes calibration and processing requirements Date Flow System Calibration Plan (CP) detailed description of all calibration procedures Date Reduction Specifications (DRS) specifies the data reduction tasks for the calibration and science data

Additional Requirements Multi-instrument support Multi-site operations Database Driven operations Extendable On the fly reprocessing

Database Requirements Persistence Object Oriented DB technology Hide schema (code is DB schema) Hide ingest/update Queries Relational DB technology Hide SQL Dynamic (new) User defined objects / tables Multi-Site Integrated bulk data storage

User Interface Requirements Developers Use pipeline API to write scripts Command Line users Recipes GUI Pipeline operations Quality Control Processing status

Pipeline components Data reduction recipes Data reduction tools Infrastructure Administration User Interface Multiple developers Multiple users Multiple sites

Development Framework Deliver usable code Working Understandable Correct

Python Object Oriented Programming Modularity Reusability Rapid Application Development Incremental development Inline documentation Understandable code Unit testing framework Working code

Unit testing Write a test for each unit of functionality To know that it works To reduce the cost of modifying code We find bugs (also in eclipse and LDAC) But, Requires discipline 25 % overhead

CVS Essential for collaborative projects Initial investment A different mind set Requires discipline Commit/Update early and often Only commit working code

Astro-Wise Pipeline - Scope In: Calibration operations defined in CP Co-added, calibrated, pointings Out: Mosaics Masking, other than CCD defects Homogenization PSF These could be covered in WP2

Astro-Wise Pipeline Prototype - Scope Reduce WFI data Baseline implementation calibration procedures Low-level data reduction routines Technology demo OO database interface Oracle9i back end Technology demo parallel processing New approach under development Establish development framework

Astro-Wise Pipeline Prototype - Status See CVS No photometric calibration yet Limited set of recipes Database back ends Parallel processing Documentation