EOVSA Pipeline Processing System J. McTiernan EOVSA Prototype Review 24-Sep-2012.

Slides:



Advertisements
Similar presentations
Welcome to the Award Winning Easiest to Use & Most Advanced View, Manage, and Control Security, Access Control, Video, Energy & Lighting Systems, & Critical.
Advertisements

Operating the Harmonizer
Peirsonpatterson,llp eDisclosure Solution ppdocs.com.
The Components There are three main components of inDepth Lite, inDepth and inDepth+ Real Time Component Reporting Package Configuration Tools.
25 July Standard and Searchable Reports. 25 July Report Types Standard Reports –“Canned” reports for the most common types of data inquiries.
OVSA Expansion Software Overview Gordon Hurford Kickoff Meeting NJIT 25-Oct-2010.
Peirsonpatterson,llp eDisclosure Solution ppdocs.com.
Dale E. Gary Professor, Physics, Center for Solar-Terrestrial Research New Jersey Institute of Technology 1 3/16/2012OVSA Preliminary Design Review.
Page 1 18-OCT-2004 RHESSI Data Analysis Workshop RHESSI Reuven Ramaty High Energy Solar Spectroscopic Imager Use or disclosure of data contained on this.
Page 1 17-Dec-2001 HESSI High Energy Solar Spectroscopic Imager Use or disclosure of data contained on this page is subject to the restrictions on the.
Software to Manage EEP Vegetation Plot Data A design proposal Michael Lee January 31, 2011.
RHESSI and OVSA Data Base Organization J. McTiernan OVSA expansion project kickoff meeting 25-Oct-2010.
FASR-A 6 Gb/s x 2 x GB/s 400 GB/day 50 GB/day ~1 GB/day X xxx FASR Data Analysis Plan 22-May-2005 Simplified Data Pipeline.
THEMIS Science Software Training1July 2007 The THEMIS Graphical User Interface J. McTiernan 23-Jul-2007
Chapter 14 Chapter 14: Server Monitoring and Optimization.
Chapter 11 - Monitoring Server Performance1 Ch. 11 – Monitoring Server Performance MIS 431 – created Spring 2006.
Page 1 18-APR-2002 RHESSI Data Analysis Workshop RHESSI Reuven Ramaty High Energy Solar Spectroscopic Imager Use or disclosure of data contained on this.
A Guide to Oracle9i1 Creating an Integrated Database Application Chapter 8.
1 QED In Vivo USB Input Output Box configuration This tutorial contains a number of instructions embedded in a great deal of explanation. Procedures that.
Online Surveys A Look at Cardiff-TeleForm Denise H. Wells Planning and Research Central Piedmont Community College.
DEMONSTRATION FOR SIGMA DATA ACQUISITION MODULES Tempatron Ltd Data Measurements Division Darwin Close Reading RG2 0TB UK T : +44 (0) F :
Financial Information System Running Reports in FIS.
Database-Driven Web Sites, Second Edition1 Chapter 8 Processing ASP.NET Web Forms and Working With Server Controls.
Detector noise analysis (DINO) demo Max Mutchler ACS-R Optimization Campaign dry-dryrun 30 January 2009.
DE&T (QuickVic) Reporting Software Overview Term
E-Cert Version 2 A Presentation by Maggie Chilton for a Chamber of Commerce.
Tutorial 1 Getting Started with Adobe Dreamweaver CS3
Rsv-control Marco Mambelli – Site Coordination meeting October 1, 2009.
Chapter 9 Scripting RMAN. Background Authors felt that scripting was a topic not covered well Authors wanted to cover both Unix/Linux and Windows environments.
A Web-based Item Bank Search, View, and Select Items for Saving, Editing, and Printing (online testing available too) - Multiple.
GLAST LAT ProjectDOE/NASA Baseline-Preliminary Design Review, January 8, 2002 K.Young 1 LAT Data Processing Facility Automatically process Level 0 data.
HIFI Tutorial 3: Getting some basic science out A.P.Marston ESAC 27 June 2013.
Marcel Casado NCAR/RAP WEATHER WARNING TOOL NCAR.
Virtual Interaction Manager
VistA Imaging Capture via Scanning. October VistA Imaging Capture via Scanning The information in this documentation includes only new and updated.
Smart Forms 2010 CAMIS Conference July 29,  Session Overview  Smart Form Process Flow  Understanding the Initial Procedures  Scan Process in.
Guide to Linux Installation and Administration, 2e1 Chapter 10 Managing System Resources.
1. To start the process, Warehouse Stationery (WSL) will invite you to use The Warehouse Group Supplier Electronic Portal and will send you the link to.
Digital Packaging Processor Gordon Hurford Jim McTiernan EOVSA PDR 15-March-2012.
EOVSA Data and Database System Gordon Hurford and Jim McTiernan EOVSA Prototype Readiness Review 24-Sept-2012.
Upgrade Process. GL Adjustments Create Backup Restore from a Backup Download & Install.
PACS NHSC Data Processing Workshop – Pasadena 10 th - 14 th Sep 2012 SPIRE AOTs, Products and Quick Look Tools Bernhard Schulz NHSC/IPAC on behalf of the.
Introduction to RHESSI Data Analysis Documentation Data Products Access to Data SSW Analysis Modes RHESSI GUI Combining GUI and Command Line RHESSI Objects.
11/25/2015Slide 1 Scripts are short programs that repeat sequences of SPSS commands. SPSS includes a computer language called Sax Basic for the creation.
Herschel Open Time Cycle 1 DP workshop ESAC, March page 1 Tour of HIFI data.
Nfs or ftp server Server 1 Server 2 ClinCapture web app SAS script SAS REST notification service [SAS RNS] (can be run as standalone app) upload / download.
EOVSA Data and Database System J. McTiernan EOVSA CoDR 25-April-2011.
Web Center Training ©2003 Optimum Technology, Inc.
Part 4 Processing and saving data with CGI/Perl Psychological Science on the Internet: Designing Web-Based Experiments From the Ground Up R. Chris Fraley.
Registration Solutions for your Event Management.
Squirrel Tutorial Pre – Pre-Process Steps Modify Single Ion (SI) M/z Calibration Fitting Baseline Fitting Donna Sueper Aerodyne, University of Colorado,
Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT.
PestPac Software. Leads The Leads Module allows you to track all of your pending sales for your company from the first contact to the close. By the end.
TROI – SPC Database Walkthrough Training Presentation Doc. USTP0213 Rev4.
Peirsonpatterson,llp eDisclosure Solution ppdocs.com.
SPI NIGHTLIES Alex Hodgkins. SPI nightlies  Build and test various software projects each night  Provide a nightlies summary page that displays all.
EOVSA EST DPP Testing J. McTiernan EOVSA Prototype Review 24-Sep-2012.
Lecture 7 Conditional Scripting and Importing/Exporting.
Merger By Dave Doulton University of Southampton.
Data Analysis GUI Documentation guide. Terminology GUI – Graphical User Interface Log-File – File containing data of interest which is generated by user’s.
03/09/2007http://pcalimonitor.cern.ch/1 Monitoring in ALICE Costin Grigoras 03/09/2007 WLCG Meeting, CHEP.
EOVSA Data and Database System J. McTiernan EOVSA Technical DesignMeeting 7-Nov-2011.
June 27-29, DC2 Software Workshop - 1 Tom Stephens GSSC Database Programmer GSSC Data Servers for DC2.
Simulation Production System Science Advisory Committee Meeting UW-Madison March 1 st -2 nd 2007 Juan Carlos Díaz Vélez.
Active-HDL Server Farm Course 11. All materials updated on: September 30, 2004 Outline 1.Introduction 2.Advantages 3.Requirements 4.Installation 5.Architecture.
Scenario use cases Szymon Mueller PSNC. Agenda 1.General description of experiment use case. 2.Detailed description of use cases: 1.Preparation for observation.
Geant4 GRID production Sangwan Kim, Vu Trong Hieu, AD At KISTI.
Bomgar Remote support software
Introduction to Computers
Presentation transcript:

EOVSA Pipeline Processing System J. McTiernan EOVSA Prototype Review 24-Sep-2012

EOVSA pipeline processing: 1)DPP – IDB: The Digital Packaging Processor (DPP) creates the Interim Database (IDB). 2) IDB: MIRIAD datasets, 5 (TBD) minutes of data per dataset (estimated 400 to 500 Mbytes per dataset if 5 minutes based on 60 Gbyte per day estimate, and 12 hours of observing time). If the root data directory is /data, then the dataset name looks like this: /data/eovsa/idb/yyyy/mm/dd/scan_id/one_idb_dataset_yyyymmdd_hhmm The IDB contains minimally-processed data, see 4 EOVSA DPP document for details.

EOVSA pipeline processing: 3) IDB – ADB: The initial pipeline jobs are scripts of mostly MIRIAD commands that access the IDB, and create Application Data Bases (ADB). For example, to create a generic ADB, a script, called eovsa_create_adb.sh runs from a cronjob every N (5?) minutes, checking for new data. (If we want to wait until night to do pipeline processing, then we can adjust the crontab file to only run the script at night.) 4) Steps in create_adb script: 1)Check to see is a process is running; a lock file, /data/eovsa/locks/create_eovsa_lock is used to determine whether the process is running (If we want to allow multiple processes, we can create multiple lock files). If the file is present, then we are done. (If the file is old (how old? TBD), then send an to alert the pipeline management personnel (or SOC, the Science Operations Center)). 2)If there is no lock file, create one. 3)Start processing. The adb results are stored in MIRIAD datasets (1 per scan?) : /data/eovsa/adb/yyyy/mm/dd/one_adb_dataset_yyyymmdd_hhmm

EOVSA pipeline processing: 5) Database Management: At each step, we create a database. For each dataset this contains (at least) the data start time, data end time and a status flag. The status flag tells us what step of processing has occurred for the given dataset. Here's a sample from RHESSI: HESSI Filedb File: Created: T10:43: Number of Files: 1196 Filename Orb_st Orb_end Start_time End_time Status_flag Npackets Drift_start Drift_end Data source hsi_ _002900_003.fits Sep-12 00:29:00 01-Sep-12 00:59: Spacecraft:vc1_ ,vc3_ hsi_ _144440_000.fits Sep-12 14:44:40 20-Sep-12 16:20: Spacecraft:vc1_ hsi_ _144440_001.fits Sep-12 14:44:40 20-Sep-12 16:20: Spacecraft:vc1_ ,vc3_ hsi_ _162000_000.fits Sep-12 16:20:00 20-Sep-12 17:55: Spacecraft:vc1_ ,vc3_ For RHESSI the status flag is 0 for files that have just been created and haven't had the next step (for RHESSI, that's qlook processing), 1 for files that have had the qlook process done, -1 for files that were overwritten with new data.

EOVSA pipeline processing: 5) Database Management (cont.): An IDB filedb entry might contain the start and end times, an identifier for the scan, and status_flag(s). We probably want to wait for completed scans before processing, so each database entry would correspond to a single scan, instead of individual IDB datasets. So the eovsa_create_adb script reads the filedb for the IDB, grabs the earliest dataset with status = 0, gets all of the data for that scan, processes, creates an ADB MIRIAD datset, sets the status flag for the IDB filedb entry to 1, creates a database entry for the new ADB dataset, with status = 0, and finishes. Rinse, repeat, for each pipeline process... 6) Ease of reprocessing: To reprocess a scan, access the filedb, set whatever status_flags = 0. We'll also provide an option for alternate reprocessing, via scan_id, which you might want to use

EOVSA pipeline processing summary: Interim DB (from DPP) Archive Process Application, File DBs Qlook Spec, FS Process Qlook Spec, Full Sun images The first Process takes the IDB; does time-dependent calibration, time and frequency averaging and creates the first of the Application databases. The next Process takes the ADB files, and creates spectrograms and full Sun images for browsing purposes and for use by the data analysis SW. The start point is the interim database, output from the DPP. This has had time-independent calibrations applied. Application databases are “Science” data, binned in time and frequency for use by the Data Analysis SW - in MIRIAD format (can also do FITS format if people want to use non-MIRIAD SW). We expect multiple datasets, magnetography, bursts, utility, type III. File databases contain information about the time range covered by each Application dataset. Quicklook Spectrograms are dynamic power spectra, functions of frequency and time, that can be used to choose time and frequency intervals by the data analysis SW. Full sun images could be used for choosing positions for high spatial resolution imaging.

Burst Process The next process examines the Qlook spectra and attempts to create a list of events. Also in this step, we look for intervals for magnetography, feeding back into the Mag. Database. Information from non-EOVSA sources may be helpful in identifying events. Event List Once we have a list of events, we go back and create more application databases, for flares, type IIIs. Event DBs The event database will be optimized in frequency and time resolution for burst studies. Event Images Images of bursts will hopefully be generated automatically, for distribution, and display on the cool EOVSA web pages. Event Archive Process Event Image Process Mag. DBModels Display

The EOVSA GUI is an IDL wrapper that will create MIRIAD scripts, which can be run locally or on a dedicated EOVSA server. The GUI borrows heavily from the RHESSI data analysis software. On initialization, the user chooses an observation time interval, and the dynamic spectrum for that time range is displayed. (Sample is FST data, but plot is simulated. ) The user will have the option of choosing time and frequency ranges graphically, or via pop-up widgets. Other settings, e.g., frequency binning, baseline choices, or frequency masks, are handled by selection widgets. Analysis SW: EOVSA IDL GUI

If you click the file pulldown: “Select Observation Time Interval” selects a time interval for which the Olook spectrogram will be plotted in the window. “Retrieve/Process Data” allows for choice of time interval and frequency bands for images and spectra. This is where most of the action is. You choose your parameters, press the “DO” button, MIRIAD commands are generated, and images or spectra are done. The rest of the buttons are for plotting, and exporting data: “Set Plot Preferences”, Sets plotting preferences for screen files and plot files, “Configure Plot File” allows the user to name the file, and output directory. “Print Plot” will print the plot, “Export Data” will allow you to export the data non- graphically, maybe in FITS format. “Exit” means exit. Currently only the Exit button is operational. The rest will be filled in.

Once the “DO” button is clicked, the IDL program generates a series of MIRIAD commands, which are written to a file. If the user has MIRIAD on his system, a process is spawned, and maps are created. If the user has no MIRIAD, then there is an option to upload the command file to our server, which will queue and then run the process. The user can choose to be notified via that his process has been run, or the IDL shell can be configured to automatically look for results. Image maps are saved in (FITS, MIRIAD or FITS or both) format. The EOVSA program will be able to read and display the image files.