1. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Increasing Tape Efficiency Original slides from HEPiX Fall 2008 Taipei RAL f2f meeting, 19/2/09Slide 1 Nicola Bessone, German Cancio, Steven Murray, Giulia Taurelli

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Contents Tape efficiency project Problem areas What has been done What is under development Roadmap and current status Slide 2

3. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Tape Efficiency Project All functionality dealing directly with storage on and management of tapes  Volume database  Migrations/recalls  Tape drive scheduling  Low-level tape positioning and read/write Team is from IT/DM Contributions from IT/FIO Slide 3

4. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Problem Areas Write more data per tape mount Use a more efficient tape format  The current tape format does not deal efficiently with small files Improve read efficiency  Require modifications from tape->disk Slide 4 Work done Current work More ideas

5. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t What has been done Slide 5

6. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Read/write More Per Mount Recall/migration policies  Freight train approach  Hold back requests based on the amount of data and elapsed time Production managers rule  Production managers plan relatively large workloads for CASTOR  Access control lists give production managers a relatively larger percentage of resources  User and group based priorities encourage users to work with their production managers Slide 6

7. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Repack: Measurements Slide 7 4 drives reading 7 drives writing 400MBytes/s Writing the current ANSI AUL format is approximately twice as slow as reading. Repack uses the cache to support asymmetric read/write drive allocation

8. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t What is under development Slide 8

9. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Writing Small Files is Slow Slide 9 Users were encouraged to store large files in Castor Unfortunately Castor contains many small files

10. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Why Small Files are Slow ANSI AUL format 3 tape marks per file 2 to 3 second per tape mark 9 seconds per data file independent of its size Slide 10 hdr1 hdr2 uh1 tm data file tm eof1 eof2 utl1 tm HeaderTrailer Tape marks 1 data file

11. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t New Tape Format Multi-file block format within the ANSI AUL format Header per block for “self description” 3 tape marks per n files n will take into account:  A configurable maximum number of files  A configurable maximum size  A configurable maximum amount of time to wait Slide 11 data file 1 … … data file n hdr1 hdr2 uh1 tm eof1 eof2 utl1 tm TrailerN data filesHeader Each 256 KB data file block written to tape includes a 1 KB header

12. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Block Header Format Slide 12 #Meta-data nameExplanationExamples Bytes for Data 1VERSION_NUMBERThe version of the block format09.135 2HEADER_SIZEHeader size in bytes010245 3 CHECKSUM_ALGORITH MName of the checksum algorithmAdler-3210 4HEADER_CHECKSUMAdler-32 checksum414688472410 5TAPE_MARK_COUNTSequential number addressing the migration-files on the tape 0000000000000001 234520 6BLOCK_SIZEBlock size in bytes inclusive of header000026214410 7BLOCK_COUNT Block offset from the beginning of the tape. Tape marks and labels are included in the count 0000000000000001 234520 8BLOCK_TIME_STAMP Time since the Epoch (00:00:00 UTC, January 1, 1970), measured in seconds122233281010 9STAGER_VERSIONThe version of the stager software2.1.7.1815 10STAGER_HOSTThe DNS name of the stager host including the domain c2cms2stager.cern.c h 30 11DRIVE_NAMEWill be provided by a local configuration file000359202810 12DRIVE_SERIALWill be provided by a local configuration file 0000000045600000 164220 13DRIVE_FIRMWAREWill be provided by a local configuration fileD3I0_C9010 14DRIVE_HOSTThe DNS name of the host including the domaintpsrv250.cern.ch30 15VOL_DENSITYThe storage capacity of the tape700.00GB10 16VOL_IDSite specific numbering system (the sticker on a tape)T0269420 17VOL_SERIALVolume Serial NumberT0269420 18DEVICE_GROUP_NAMEThe device group name that linked the tape to the drive3592B110 19FILE_SIZEThe size of the data file in bytes 0000000109951162 777620 FILE_CHECKSUMAdler-32 checksum192686061610 21FILE_NS_HOSTThe DNS name of the host including the domaincastorns.cern.ch30 22FILE_NS_IDThe name server ID of the data file22699427420 23 FILE_PROGESSIVE_CH ECKSUM Adler-32. Progressive checksum of all the blocks written to tape so far for the current data file123456789010 24FILE_BLOCK_COUNTBlock offset from the beginning of the data file 0000000000000001 234520 Header size before file_name :375 25FILE_NAME Last “x” bytes of the filename from the name server. This field acts as a padding to the nearest KiB.649 Header size :1024 VERSION_NUMBER HEADER_SIZE CHECKSUM_ALGORITHM HEADER_CHECKSUM TAPE_MARK_COUNT BLOCK_SIZE BLOCK_COUNT BLOCK_TIME_STAMP STAGER_VERSION STAGER_HOST DRIVE_NAME DRIVE_SERIAL DRIVE_FIRMWARE DRIVE_HOST VOL_DENSITY VOL_ID VOL_SERIAL DEVICE_GROUP_NAME FILE_SIZE FILE_CHECKSUM FILE_NS_HOST FILE_NS_ID FILE_PROGESSIVE_CHECKSUM FILE_BLOCK_COUNT FILE_NAME

13. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Predicted Tape Format Performance Slide 13 New tape formatAUL tape format Less than 1 year with the new tape format Greater than 4 years with the AUL tape format Hardware manufactures increasing tape density impose a 2 year cycle The future of repack may be to run continuously to verify checksums and ensure data integrity

14. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Architecture Needs to Change The new tape format is only half of the story An aggregator needs to be inserted into the disk ↔ tape data streams Anything old that is replaced is an opportunity for code re-use and increased maintainability via the Castor framework Slide 14

15. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Slide 15 Drive Scheduler Drive Scheduler Tape Server Tape Server Disk Server Disk Server Disk Server Disk Server Disk Server Disk Server 12 1. Stager requests a drive 2. Drive is allocated 3. Data is transferred to/from disk/tape based on file list given by stager 3 3 Legend Data Control messages Host Stager Current Architecture 1 data file = 1 tape file

16. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Stager New Architecture Slide 16 Drive Scheduler Drive Scheduler Tape Server Disk Server Disk Server Disk Server Disk Server Disk Server Disk Server Stager Legend Data to be stored Control messages Host Server process(es) Tape Gateway Tape Aggregator n data files = 1 tape file The tape gateway will replace RTCPClientD The tape gateway will be stateless The tape aggregator will wrap RTCPD

17. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Roadmap Slide 17 DateActions Beginning Q4 2008Put repack into full production will at least 20 drives. Expecting 700 MB/s. Conclude new tape format architecture. End Q1 2009Release first functional prototype of new tape format. End Q2 2009Write new tape format with repack only. Read new tape format everywhere. End Q3 2009Read and write everywhere Beginning Q1 2010Replace RTCPD with tape aggregator Current status (Feb 2009):  New tape format specified  Classes for reading/writing new format coded  Communication between rtcpclientd/rtcpd/VDQM reverse engineered and understood  Tape aggregator and gateway prototypes underway; first goal to fully support non- aggregated AUL read/writes; aggregations to come later