Presentation is loading. Please wait.

Presentation is loading. Please wait.

CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Tape-dev update Castor F2F meeting, 14/10/09 Nicola Bessone, German Cancio, Steven Murray,

Published byNorah Fox Modified over 9 years ago

Similar presentations

Presentation on theme: "CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Tape-dev update Castor F2F meeting, 14/10/09 Nicola Bessone, German Cancio, Steven Murray,"— Presentation transcript:

1 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Tape-dev update Castor F2F meeting, 14/10/09 Nicola Bessone, German Cancio, Steven Murray, Giulia Taurelli

2 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Slide 2 Drive Scheduler Drive Scheduler Tape Server Tape Server Disk Server Disk Server Disk Server Disk Server Disk Server Disk Server 12 1. Stager requests a drive 2. Drive is allocated 3. Data is transferred to/from disk/tape based on file list given by stager 3 3 Legend Data Control messages Host Stager Current Architecture 1 data file = 1 tape file Reminder – last F2F

3 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Stager New Architecture Slide 3 Drive Scheduler Drive Scheduler Tape Server Disk Server Disk Server Disk Server Disk Server Disk Server Disk Server Stager Legend Data to be stored Control messages Host Server process(es) Tape Gateway Tape Aggregator n data files = 1 tape file The tape gateway will replace RTCPClientD The tape gateway will be stateless The tape aggregator will wrap RTCPD Reminder – last F2F

4 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t New software Goals: code refresh (unmaintained/unknown), component reuse (Castor C++ / DB framework), improved (DB) consistency, enhanced stability -> performance, ground work for future new tape format (block-based metadata) 2 new daemons developed:  tapegatewayd (on the stager) -> replaces rtcpclientd / recaller / migrator.  aggregatord (on the tape server) -> acts as a proxy or bridge between rtcpd and tapegatewayd. (No new tape format yet) Rewritten migHunter  Transactional handling (at stager DB level) of new migration candidates German.Cancio@cern.ch Slide 4

5 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Status software has been installed on CERN’s stress test instance (ITDC) ~4w ago, started end-to-end tests and stress tests (~20 tape servers, ~25 disk servers) So far, significant improvements in terms of stability (no software- related tape unmounts during migrations and recalls) However: testing not completed yet, many issues found on the way unveiled by the new software  See next slides New migHunter to be released ASAP (2.1.9-2 if tests with rtcpclientd ok) Tape gateway + aggregator to be released in 2.1.9-x as optional component - not part of the default deployment, and there are no dependencies on it from the rest of the CASTOR software. German.Cancio@cern.ch Slide 5

6 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Test findings (1) Performance degradations during migrations  Already observed in production, but difficult to trace down as long-lived migration streams rarely occur (cf savannah)  Found to be a misconfiguration in the rtcpd / syslog config, causing log messages to be generated growing @ O(n*n),, n=migrated files Another problem to be understood is stager DB time for disk server/ fs selection, which tends to grow during migration lifetime. Currently not limited by this but could become a bottleneck Slide 6

7 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Test findings (2) Migration slowdown on IBM drives  Castor at fault? Towards end of tape? End of mount? German.Cancio@cern.ch Slide 7

8 Tpsrv150, 23/9/09 Tpsrv151, 23/9/09 Tpsrv001, 23/9/09 Tpsrv235, 23/9/09 Tpsrv204, 23/9/09 Tpsrv203, 24/9/09 Tpsrv204, 24/9/09

9 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Test findings (2) Migration slowdown on IBM drives  Castor at fault? Towards end of tape? End of mount?  correlation between where the tape is being written and performance of writing. Confirmed by writing a Castor- independent test writing Castor-like AUL files  Traced down to be an IBM hardware specific issue. After analysis, TapeOps confirmed this to be part of an optimisation on IBM drives called “virtual back hitch”. This optimisation allows small files to be written at higher speeds by reserving a special cache area on tape, while the tape is not getting full.  NVC can be switched off, but performance drops to ~15MB/s German.Cancio@cern.chslide 9

10 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Test findings (3) Under (yet) unknown circumstances, IBM tapes hit end-of- tape at 10-30% less than their nominal capacity. Read performance on these tapes is also suboptimal Seems to be related to a suboptimal working of NVC / virtual back hitch  Does not occur when NVC is switched off To be reported to IBM German.Cancio@cern.ch Slide 10 reading tape with urandom- generated 100MB files to /dev/null using dd (X: seconds, Y: MB/s throughput). The tape contains 8222 AUL files of 100M each

11 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Test findings (4) Suboptimal file placement strategy on recalls?  which apparently causes interference Recall using default Castor file placement Same recall using 2 dedicated disk servers per tape server German.Cancio@cern.chSlide 11 3 tape servers recalling on 7 disk servers (all files distributed over all disk servers/file systems 3 tape and 6 disk servers (all filesystems), same as above yields ~310-320 MB/s

12 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Test findings (5) Recall performance limited by central element (gateway/stager/..?)  a central limitation which prevents performance to go higher than a threshold, even if distinct pools are being used German.Cancio@cern.chSlide 12 c2itdc total throughput c2itdc/ pool 1 c2itdc / pool 2 shortly after 21:30, the tape recall on pool 1finished. recall performance of the second pool goes up from then on, and that the total recall performance (both disk pools) stays at ~255MB/s. No DB / network contention.

13 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Test findings (7) Performance degradation on recalls on new tape server HW  observed that new-generation tape servers (Dell 4core) are capable to read out data from tape at a higher than rtcpd is capable to process it. This eventually causes the attached drives to stall. It happens equally if an IBM or an STK drive is attached. The stalling problem does not happen on all other older servers (elonex 2core, clustervision) as there, the drives read out at lower speeds.  Traced down (yesterday..) to a too verbose logging of the tape positioning executable (posovl) when using the new syslog-based DLF. German.Cancio@cern.chSlide 13,

14 CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t “tape” bug fixes in 2.1.9 “tape” = repack, VDQM, VMGR, rcpclientd, rtcpd, taped, and the new components 2.1.9-0  https://twiki.cern.ch/twiki/bin/view/DataManagement/Cast orReleasePlan21900 https://twiki.cern.ch/twiki/bin/view/DataManagement/Cast orReleasePlan21900 2.1.9-2 (planned)  https://twiki.cern.ch/twiki/bin/view/DataManagement/Cast orReleasePlan21902 https://twiki.cern.ch/twiki/bin/view/DataManagement/Cast orReleasePlan21902 2.1.9-X  https://twiki.cern.ch/twiki/bin/view/DataManagement/Cast orTapeReleasePlan219X https://twiki.cern.ch/twiki/bin/view/DataManagement/Cast orTapeReleasePlan219X German.Cancio@cern.chSlide 14

Download ppt "CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Tape-dev update Castor F2F meeting, 14/10/09 Nicola Bessone, German Cancio, Steven Murray,"

Similar presentations

1 Magnetic Disks 1956: IBM (RAMAC) first disk drive 5 Mb – 0.002 Mb/in2 35000$/year 9 Kb/sec 1980: SEAGATE first 5.25’’ disk drive 5 Mb – 1.96 Mb/in2 625.

1 Magnetic Disks 1956: IBM (RAMAC) first disk drive 5 Mb – Mb/in $/year 9 Kb/sec 1980: SEAGATE first 5.25’’ disk drive 5 Mb – 1.96 Mb/in2 625.
Distributed Storage March 12, 2001. 2 Distributed Storage What is Distributed Storage?  Simple answer: Storage that can be shared throughout a network.

Distributed Storage March 12, Distributed Storage What is Distributed Storage?  Simple answer: Storage that can be shared throughout a network.
16/9/2004Features of the new CASTOR1 Alice offline week, 16/9/2004 Olof Bärring, CERN.

16/9/2004Features of the new CASTOR1 Alice offline week, 16/9/2004 Olof Bärring, CERN.
CASTOR Project Status CASTOR Project Status CERNIT-PDP/DM February 2000.

CASTOR Project Status CASTOR Project Status CERNIT-PDP/DM February 2000.
CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Oracle and Streams Diagnostics and Monitoring Eva Dafonte Pérez Florbela Tique Aires.

CERN - IT Department CH-1211 Genève 23 Switzerland t Oracle and Streams Diagnostics and Monitoring Eva Dafonte Pérez Florbela Tique Aires.
Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t DSS New tape server software Status and plans CASTOR face-to-face.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS New tape server software Status and plans CASTOR face-to-face.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Tier0 Status Tony Cass (With thanks to Miguel Coelho dos Santos & Alex Iribarren) LCG-LHCC.

CERN IT Department CH-1211 Genève 23 Switzerland t Tier0 Status Tony Cass (With thanks to Miguel Coelho dos Santos & Alex Iribarren) LCG-LHCC.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Plans and Architectural Options for Physics Data Analysis at CERN D. Duellmann, A. Pace.

CERN IT Department CH-1211 Genève 23 Switzerland t Plans and Architectural Options for Physics Data Analysis at CERN D. Duellmann, A. Pace.
Building Advanced Storage Environment Cheng Yaodong Computing Center, IHEP December 2002.

Building Advanced Storage Environment Cheng Yaodong Computing Center, IHEP December 2002.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Experience with Windows Vista at CERN Rafal Otto Internet Services Group IT Department.

CERN IT Department CH-1211 Genève 23 Switzerland t Experience with Windows Vista at CERN Rafal Otto Internet Services Group IT Department.
Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t DSS Summary of CASTOR incident, April 2010 Germán Cancio Leader,

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS Summary of CASTOR incident, April 2010 Germán Cancio Leader,
DMC Technology Server Re-fresh. Contents Evaluate vendor Blade technology Develop a Blade architecture that meets DMC/PBB requirements Develop a stand-alone.

DMC Technology Server Re-fresh. Contents Evaluate vendor Blade technology Develop a Blade architecture that meets DMC/PBB requirements Develop a stand-alone.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Data Management Evolution and Strategy at CERN G. Cancio, D. Duellmann, A. Pace With input.

CERN IT Department CH-1211 Genève 23 Switzerland t Data Management Evolution and Strategy at CERN G. Cancio, D. Duellmann, A. Pace With input.
CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t Storageware Flavia Donno CERN WLCG Collaboration Workshop CERN, 13-14 November 2008.

CERN IT Department CH-1211 Geneva 23 Switzerland t Storageware Flavia Donno CERN WLCG Collaboration Workshop CERN, November 2008.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Tier0 Status - 1 Tier0 Status Tony Cass LCG-LHCC Referees Meeting 18 th November 2008.

CERN IT Department CH-1211 Genève 23 Switzerland t Tier0 Status - 1 Tier0 Status Tony Cass LCG-LHCC Referees Meeting 18 th November 2008.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Castor development status Alberto Pace LCG-LHCC Referees Meeting, May 5 th, 2008 DRAFT.

CERN IT Department CH-1211 Genève 23 Switzerland t Castor development status Alberto Pace LCG-LHCC Referees Meeting, May 5 th, 2008 DRAFT.
Functional description Detailed view of the system Status and features Castor Readiness Review – June 2006 Giuseppe Lo Presti, Olof Bärring CERN / IT.

Functional description Detailed view of the system Status and features Castor Readiness Review – June 2006 Giuseppe Lo Presti, Olof Bärring CERN / IT.
CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/it Castor External Operation Face-to-Face Meeting, CNAF, October 29-31, 2007 CASTOR2 Disk.

CERN - IT Department CH-1211 Genève 23 Switzerland Castor External Operation Face-to-Face Meeting, CNAF, October 29-31, 2007 CASTOR2 Disk.
Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t DSS Tape Monitoring Vladimír Bahyl IT DSS TAB Storage Analytics.

Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS Tape Monitoring Vladimír Bahyl IT DSS TAB Storage Analytics.
McLean 20061 HIGHER COMPUTER NETWORKING Lesson 15 (a) Disaster Avoidance Description of disaster avoidance: use of anti-virus software use of fault tolerance.

McLean HIGHER COMPUTER NETWORKING Lesson 15 (a) Disaster Avoidance Description of disaster avoidance: use of anti-virus software use of fault tolerance.

Similar presentations

Ads by Google