Advances in Bit Preservation (since DPHEP’2015) 3/2/2016 DPHEP / WLCG Workshop1 Germán Cancio IT Storage Group CERN DPHEP / WLCG Workshop Lisbon, 3/2/2016.

Slides:

Advertisements

Similar presentations

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS Bit preservation cost outlook: Cost for 10, 20, 30 years archive.

Advertisements

XenData SX-520 LTO Archive Servers A series of archive servers based on IT standards, designed for the demanding requirements of the media and entertainment.

© 2011 IBM Corporation1 Tape Logical Block Protection (aka Tape Checksum ) Kevin D. Butt SCSI Architect, Data Protection & Retention, IBM T10/SSC Working.

WS2012 File System Enhancements: ReFS and Storage Spaces Rick Claus Sr. Technical WSV316.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS TSM CERN Daniele Francesco Kruse CERN IT/DSS.

Copyright 2014 Kenneth M. Chipps Ph.D. Network Management Using Sensors to Monitor Network Equipment Rooms Last Update

CASTOR Project Status CASTOR Project Status CERNIT-PDP/DM February 2000.

Hugo HEPiX Fall 2005 Testing High Performance Tape Drives HEPiX FALL 2005 Data Services Section.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS Update on CERN Tape Status HEPiX Spring 2014, Annecy German.

© 2009 IBM Corporation Statements of IBM future plans and directions are provided for information purposes only. Plans and direction are subject to change.

Agenda  Overview  Configuring the database for basic Backup and Recovery  Backing up your database  Restore and Recovery Operations  Managing your.

AS Level ICT Selection and use of storage requirements, media, and devices: Devices and media.

1 RAL Status and Plans Carmine Cioffi Database Administrator and Developer 3D Workshop, CERN, November 2009.

Oracle Database High Availability Brandon Kuschel Jian Liu Source: Oracle Database 11g Release 2 High Availability An Oracle White Paper November 2010.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS New tape server software Status and plans CASTOR face-to-face.

Experiences and Challenges running CERN's High-Capacity Tape Archive 14/4/2015 CHEP 2015, Okinawa2 Germán Cancio, Vladimír Bahyl

Operating in a SAN Environment March 19, 2002 Chuck Kinne AT&T Labs Technology Consultant.

CERN IT Department CH-1211 Genève 23 Switzerland t Tape-dev update Castor F2F meeting, 14/10/09 Nicola Bessone, German Cancio, Steven Murray,

Chapter 7 Making Backups with RMAN. Objectives Explain backup sets and image copies RMAN Backup modes’ Types of files backed up Backup destinations Specifying.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS Summary of CASTOR incident, April 2010 Germán Cancio Leader,

ATLAS Metrics for CCRC’08 Database Milestones WLCG CCRC'08 Post-Mortem Workshop CERN, Geneva, Switzerland June 12-13, 2008 Alexandre Vaniachine.

HEPiX bit-preservation WG update – Spring 2014 Dmitry Ozerov/DESY Germán Cancio/CERN HEPiX Spring 2014, Annecy.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS (Physics) Archival Storage Status and Experiences at CERN.

CERN - IT Department CH-1211 Genève 23 Switzerland The Tier-0 Road to LHC Data Taking CPU ServersDisk ServersNetwork FabricTape Drives.

Update on Bit Preservation, HEPiX WG and Beyond 8/6/2015 DPHEP Collaboration Workshop1 Germán Cancio, IT-DSS-TAB CERN DPHEP Collaboration Workshop CERN,

 CASTORFS web page - CASTOR web site - FUSE web site -

Ian Bird Trigger, Online, Offline Computing Workshop CERN, 5 th September 2014.

Report from CASTOR external operations F2F meeting held at RAL in February Barbara Martelli INFN - CNAF.

CERN - IT Department CH-1211 Genève 23 Switzerland t Oracle Real Application Clusters (RAC) Techniques for implementing & running robust.

Your university or experiment logo here The Protocol Zoo A Site Presepective Shaun de Witt, STFC (RAL)

CERN - IT Department CH-1211 Genève 23 Switzerland t High Availability Databases based on Oracle 10g RAC on Linux WLCG Tier2 Tutorials, CERN,

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS New tape server software Status and plans CASTOR face-to-face.

CASTOR project status CASTOR project status CERNIT-PDP/DM October 1999.

01. December 2004Bernd Panzer-Steindel, CERN/IT1 Tape Storage Issues Bernd Panzer-Steindel LCG Fabric Area Manager CERN/IT.

CERN IT Department CH-1211 Genève 23 Switzerland t The Tape Service at CERN Vladimír Bahyl IT-FIO-TSI June 2009.

Preservation e-Infrastructures, Certification & ADMP IGs DPHEP Status and Outlook RDA Plenary 6 Paris, September 2016 International.

International Collaboration for Data Preservation and Long Term Analysis in High Energy Physics RECODE - Final Workshop - January.

CERN IT Department CH-1211 Genève 23 Switzerland t Increasing Tape Efficiency Original slides from HEPiX Fall 2008 Taipei RAL f2f meeting,

Tape write efficiency improvements in CASTOR Department CERN IT CERN IT Department CH-1211 Genève 23 Switzerland DSS Data Storage.

Tape archive challenges when approaching Exabyte-scale CHEP 2010, Taipei G. Cancio, V. Bahyl, G. Lo Re, S. Murray, E. Cano, G. Lee, V. Kotlyar CERN IT-DSS.

Preparing Data Management Plans for WLCG and HNISciCloud IT International Collaboration for Data Preservation and Long Term.

Storage & Database Team Activity Report INFN CNAF,

Usecases: 1.ISIS Neutron Source 2.DP for HEP Matthew Viljoen STFC, UK APARSEN-EGI workshop: preserving big data for research Amsterdam Science Park 4-6.

TEVATRON DATA CURATION UPDATE Gene Oleynik, Fermilab Department Head, Data Movement and Storage 1.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS CASTOR and EOS status and plans Giuseppe Lo Presti on behalf.

School on Grid & Cloud Computing International Collaboration for Data Preservation and Long Term Analysis in High Energy Physics.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS CASTOR status and development HEPiX Spring 2011, 4 th May.

CTA: CERN Tape Archive Rationale, Architecture and Status

CERN IT-Storage Strategy Outlook Alberto Pace, Luca Mascetti, Julien Leduc

Oracle Database High Availability

Large-scale Archival Storage - a brief overview for the HEP use case -

Integrating Disk into Backup for Faster Restores

Tape Drive Testing IBM 3592.

Robotics and Tape Drives

EOSCpilot WP4: Use Case 5 Material for

Tape Drive Testing.

Tape Operations Vladimír Bahyl on behalf of IT-DSS-TAB

The Unbearable Slowness of Tape

Experiences and Outlook Data Preservation and Long Term Analysis

Test C : IBM Enterprise Storage Technical Support V5

CTA: CERN Tape Archive Adding front-ends and back-ends Status report

Oracle Database High Availability

Pierre-Emmanuel Brinette

CERN Site Report Giuseppe Lo Presti

Ákos Frohner EGEE'08 September 2008

CTA: CERN Tape Archive Overview and architecture

Technology for Long Term Digital Preservation Workshop ESA 22/09/2017

OffLine Physics Computing

Tape Portfolio Messaging

Presentation transcript:

Advances in Bit Preservation (since DPHEP’2015) 3/2/2016 DPHEP / WLCG Workshop1 Germán Cancio IT Storage Group CERN DPHEP / WLCG Workshop Lisbon, 3/2/2016

Outline Advances at CERN since last DPHEP WS Environmental sensor Logical Block Protection LEP data on EOS Outlook for /2/2016 DPHEP / WLCG Workshop

Environmental sensor (aka ”dust sensor”) 3 3/2/2016 DPHEP / WLCG Workshop

Environmental sensor (aka ”dust sensor”) 4 3/2/2016 DPHEP / WLCG Workshop Sensors in full production at CERN Dust Temperature Relative humidity Specs available via ohwr.orgohwr.org HW design & Arduino board schematics Rpi software Puppet templates Can be integrated in tape libraries or used stand-alone Presented at Oracle LTUG Interest from other sites and vendor

Environmental sensor (aka ”dust sensor”) 5 3/2/2016 DPHEP / WLCG Workshop

Logical Block Protection 6 3/2/2016 DPHEP / WLCG Workshop

Available in latest CASTOR release, deployed at CERN Support for IBM and Oracle enterprise drives (using crc32c) (small changes required to make it work on LTO as well) Blocks checksummed and verified during both read and write operations Low overhead – max 5% for writing, zero for reading Next step: stand-alone tape verification without sending any data off the drive, working at full streaming speed 7 3/2/2016 DPHEP / WLCG Workshop Logical Block Protection

Other improvements Extended tape verification Light mode – after every write mount, verify critical tape areas (BOT, EOT, random sample in the middle) Exploit low-level tape system information Transient/internal drive read/write/mount stats at SCSI level; library low-level logs Assess the state of the drive and forecast a potential failure before it actually happens Differences between Oracle and IBM – needs homogenization 8 3/2/2016 DPHEP / WLCG Workshop T05:46: :00 tpsrv220 tapeserverd[3335]: LVL=Info TID=3350 MSG="Logging volume statistics" firmwareVersion="460E" lifetimeBOTPasses="1486" lifetimeMOTPasses="1556" lifetimeVolumeMounts="202" lifetimeVolumeRecoveredReadErrors="167" lifetimeVolumeRecoveredWriteErrors="30" lifetimeVolumeUnrecoveredReadErrors="4" lifetimeVolumeUnrecoveredWriteErrors="2" validity="1" volumeManufacturingDate=" " T05:46: :00 tpsrv220 tapeserverd[3335]: LVL=Info TID=3350 MSG="Logging volume statistics" firmwareVersion="460E" lifetimeBOTPasses="1486" lifetimeMOTPasses="1556" lifetimeVolumeMounts="202" lifetimeVolumeRecoveredReadErrors="167" lifetimeVolumeRecoveredWriteErrors="30" lifetimeVolumeUnrecoveredReadErrors="4" lifetimeVolumeUnrecoveredWriteErrors="2" validity="1" volumeManufacturingDate=" " Not good.. Really bad!

EOS 9 3/2/2016 DPHEP / WLCG Workshop timeline to be defined in 2016

Outlook for Expect +45PB/year in LS2 ( ) ideal moment in time for next repack (media replacement) ~260PB to repack, compared to 85PB in 2014/15 However, new drive generation in ~2017 may allow media reuse … and significant $aving$ Another 160PB to move! After repack is before repack! 10 3/2/2016 DPHEP / WLCG Workshop