Nov 26 2012 HST Data Processing Operations and Engineering Division Data Systems Branch Mark Kyprianou.

Slides:

Advertisements

Similar presentations

Dec 2, 2014 The HST Online Cache: Changes to the User Experience Karen Levay Faith Abney Mark Kyprianou.

Advertisements

Agile Infrastructure built on OpenStack Building The Next Generation Data Center with OpenStack John Griffith, Senior Software Engineer,

Mission Engineering and Science Analysis (MESA) Jeff Valenti What is MESA?

Provenance in Open Distributed Information Systems Syed Imran Jami PhD Candidate FAST-NU.

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.

Overview Of Microsoft New Technology ENTER. Processing....

Building a Framework for Data Preservation of Large-Scale Astronomical Data ADASS London, UK September 23-26, 2007 Jeffrey Kantor (LSST Corporation), Ray.

Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.

© InLoox ® InLoox PM Web App product presentation The Online Project Software.

System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.

Microsoft ® Application Virtualization 4.6 Infrastructure Planning and Design Published: September 2008 Updated: February 2010.

Nov Introduction and Highlights Rick White Karen Levay.

Upcoming Enhancements to the HST Archive Mark Kyprianou Operations and Engineering Division Data System Branch.

Lecture 8 – Platform as a Service. Introduction We have discussed the SPI model of Cloud Computing – IaaS – PaaS – SaaS.

Technology Overview. Agenda What’s New and Better in Windows Server 2003? Why Upgrade to Windows Server 2003 ?  From Windows NT 4.0  From Windows 2000.

Networked Application Architecture Design. Application Building Blocks Application Software Data Infrastructure Software Local Area Network Server Desktop.

Data Management Subsystem: Data Processing, Calibration and Archive Systems for JWST with implications for HST Gretchen Greene & Perry Greenfield.

Preparing your Fabric & Apps for Windows Server 2003 End of Support Jeff Woolsey Principal Program Manager.

SPACE TELESCOPE SCIENCE INSTITUTE Operated for NASA by AURA COS Pipeline Language(s) We plan to develop CALCOS using Python and C Another programming language?

GCMD/IDN STATUS AND PLANS Stephen Wharton CWIC Meeting February19, 2015.

Data Management Subsystem Jeff Valenti (STScI). DMS Context PRDS - Project Reference Database PPS - Proposal and Planning OSS - Operations Scripts FOS.

2-1 DMS Requirements Status Daryl Swade DMS Systems Engineer Nov. 6, 2013 S&OC Data Management System Design Review 3.

Cloud Computing in NASA Missions Dan Whorton CTO, Stinger Ghaffarian Technologies June 25, 2010 All material in RED will be updated.

Amazon Web Services BY, RAJESH KANDEPU. Introduction  Amazon Web Services is a collection of remote computing services that together make up a cloud.

Presentation To. Mission Think Dynamics is in the business of automating the management of data center resources thereby enabling senior IT executives.

Implementation Review1 Deriving Architecture Requirements March 14, 2003.

From Virtualization Management to Private Cloud with SCVMM 2012 Dan Stolts Sr. IT Pro Evangelist Microsoft Corporation

OOI CI LCA REVIEW August 2010 Ocean Observatories Initiative OOI Cyberinfrastructure Architecture Overview Michael Meisinger Life Cycle Architecture Review.

ALICE Upgrade for Run3: Computing HL-LHC Trigger, Online and Offline Computing Working Group Topical Workshop Sep 5 th 2014.

How to Improve Operational Efficiencies with an Enterprise Document Management Solution.

And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR

Nov JWST Data Management Systems Gretchen Greene.

Cloud Computing Ed Lazowska Bill & Melinda Gates Chair in Computer Science & Engineering University of Washington.

Afresco Overview Document management and share

Biomedical Informatics Research Network BIRN Workflow Portal.

How Are Customers Using Opalis? Automate provisioning, resource allocation and retirement Extend virtual machine management to the cloud Virtual Machine.

Evolution of the JPSS Ground Project Calibration and Validation System Patrick Purcell, Gyanesh Chander and Peyush Jain JPSS Ground Project NASA, GSFC.

Big Data to Knowledge Panel SKG 2014 Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China August Geoffrey Fox

Implementation Review1 Archive Ingest Redesign March 14, 2003.

23-Nov-1999STScI Projects Monthly Status Review1 of 8 SpaceTelescopeScienceInstitute COS Status Report for period December, 1998 to November, 1999 Tony.

Copyright c 2004 OSIsoft Inc. All rights reserved. By Mark Hughes OSIsoft, Inc. Update on Status and Directions.

1 Future Directions in HST Data Processing 19 November 2004.

Gerhard Dueck -- CS3013Architecture 1 Architecture-Centric Process  There is more to software development then going blindly through the workflows driven.

Cloud Computing from a Developer’s Perspective Shlomo Swidler CTO & Founder mydrifts.com 25 January 2009.

Cloud, big data, and mobility Your phone today probably meets the minimum requirements to run Windows Server 2003 Transformational change up.

FlexPod Converged Solution. FlexPod is… A prevalidated flexible, unified platform featuring: Cisco Unified Computing System™ Programmable infrastructure.

Cofax Scalability Document Version Scaling Cofax in General The scalability of Cofax is directly related to the system software, hardware and network.

Red Hat, Inc. The Revolution of Choice. Red Hat, Inc. Founded in 1995 –Bob Young, CEO - Co-founder –Marc Ewing, CTO - Co-founder Headquartered in Research.

Microsoft Virtual Academy Module 9 Configuring and Managing the VMM Library.

File system: Ceph Felipe León fi Computing, Clusters, Grids & Clouds Professor Andrey Y. Shevel ITMO University.

Comparison of VPS Hosting and Cloud Hosting Features.

Clouding with Microsoft Azure

HST and JWST Pipelines and Reference Files

Fundamentals of Information Systems, Sixth Edition

Let's talk about Linux and Virtualization in 'vLAMP'

Computing Clusters, Grids and Clouds Globus data service

WEB APPLICATION Diagram Template

DocFusion 365 Intelligent Template Designer and Document Generation Engine on Azure Enables Your Team to Increase Productivity MICROSOFT AZURE APP BUILDER.

Medical Data Manager use case: 3D medical images analysis workflow.

Joseph JaJa, Mike Smorul, and Sangchul Song

Hadoop Clusters Tess Fulkerson.

Overview Introduction VPS Understanding VPS Architecture

20409A 7: Installing and Configuring System Center 2012 R2 Virtual Machine Manager Module 7 Installing and Configuring System Center 2012 R2 Virtual.

InLoox PM Web App product presentation

SDM workshop Strawman report History and Progress and Goal.

Data Systems Environment at SM4

Managing Work in the New Computing Environment March 14, 2003

Client/Server Computing and Web Technologies

Johan Lindberg, inRiver

Windows Azure SDK 1.7 and New Features

Presentation transcript:

Nov HST Data Processing Operations and Engineering Division Data Systems Branch Mark Kyprianou

Nov Highlights - Infrastructure Deployed database clustering technologies to DADS/OPUS systems – Internal and Public databases – Improve uptime, reliability and performance Updated/added new web servers – 3 independent environments: Dev, Test and OPS Allows for isolation in the software life cycle

Nov Highlights - Instruments Science data processing enhancements – Astrodrizzle (ACS and WFC3) – CALACS w/CTE – COS reprocessing/cataloging enhancements – CALCOS with no IRAF – STIS flight software problem with time tag data – STIS Reprocessing to create valid CCI files for post-SM4 data

Nov Retrieval Times

Nov Future Development

Nov Future Development Infrastructure – Red Hat OS upgrade for DADS/OPUS servers – Purchased NAS storage (EMC Isilon) Very scalable for Big Data Blades support – High IOPS – CPU processing – Storage

Nov Future Development Instrument Data Processing enhancements – Keywords – Algorithmic enhancements Reprocessing planned for ACS, WFC3 and COS Iraf Free: WFC3 and STIS – In test

Nov HST Upgrade Projects

Nov HST Upgrade Projects Background Automated Reprocessing (BAR) – Integrated reprocessing – Online cache Workflow Manager (WFM) – Technology refresh for work flow processing Calibration Reference Data System (CRDS) Driven by analysis of current systems targeting JWST HST Mission Office elected to adopt these enhancements

Nov Background Automated Reprocessing (BAR) Handle reprocessing activities in an integrated fashion – Use prioritization scheme to control processing sequence Provide online access to files Enhance tiered file storage sub-system – Store data to provide direct access – Rework file handling to be mission independent

Nov Work Flow Manager (WFM) Upgrade workflow processing – OPUS technology refresh is over due – Limited functionality for expansion (cloud computing, additional compute resources) Condor: Open source, high-throughput computing software framework has been selected – Allows to access compute resources on different levels/rules

Nov Overview: BAR/WFM/CRDS

Nov Current Status BAR – SDPQ - complete – File Storage to online cache – in progress – Working on distribution updates WFM – Condor Enterprise level, Open source workflow package Open Workflow Language (OWL) – Blackboard, template for data centric processing – Shoveler - complete – Prototype CALACS pipeline Continued development thru FY13

Nov Questions/Comments