1. Developments for tape CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t DSS Developments for tape CASTOR workshop 2012 Author: Steven Murray (steven.murray@cern.ch)

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services DSS Developments for tape - 2 Contents Immediate tape marks Patching the SLC6 st driver Configuring the bulk interface of tape The solution to UNKNOWN tape drives Virtual tape library - mhvtl

3. Data & Storage Services Before immediate tape marks Developments for tape - 3 Legacy tape transfer-manager RTCPCLIENTD Drive scheduler VDQM Legacy tape reader/writer RTCPD 1. Mount tape 2. File info 3. Write header file 4. Flush buffer 5. Write user file 6. Flush buffer 7. Write trailer file 8. Flush buffer 9. Wrote file 3 flushes per user file ≈ 5 seconds ≈ 1.2 GB that could have been written

4. Data & Storage Services What is an immediate tape mark Synchronous tape mark - Mark the end of file and flush struct mtop mtop; mtop.mt_op = MTWEOF; mtop.mt_count = 1; ioctl(tapefd, MTIOCTOP, &mtop); Immediate tape mark – Just mark the end of file; no flush struct mtop mtop; mtop.mt_op = MTWEOFI; mtop.mt_count = 1; ioctl(tapefd, MTIOCTOP, &mtop); Flush tape drive buffer - Flush struct mtop mtop; mtop.mt_op = MTWEOF; mtop.mt_count = 0; ioctl(tapefd, MTIOCTOP, &mtop); Developments for tape - 4

5. Data & Storage Services After immediate tape marks Developments for tape - 5 Drive scheduler VDQM 1. Mount tape Tape gatewa y 2. File info for N files For N files or data loop 3. Write header file 4. Write user file 5. Write trailer file End loop 6. Flush buffer 7. Wrote N files Protocol bridge Legacy tape reader/writer RTCPD Legacy tape transfer-manager RTCPCLIENTD Tape format has not been changed! Bulk protocol Drive buffer flushed less often

6. Data & Storage Services Results of immediate tape marks Developments for tape - 6

7. Data & Storage Services SLC5 st driver has 2 patches MTWEOFI –Provides support for immediate tape marks –CASTOR cannot run without it MTIOCSENSE –Improves error diagnosis –CASTOR can run without it. Developments for tape - 7

8. Data & Storage Services SLC6 st driver has 1 patch The SLC6 st driver currently has the MTWEOFI patch The SLC6 st driver is used by virtual tape servers Real hardware will decide whether or not we require the MTIOCSENSE patch Developments for tape - 8

9. Data & Storage Services Tape bulk-interface configuration TAPEBRIDGE BULKREQUESTMIGRATIONMAXBYTES –The maximum number of bytes a set of files to migrate should represent TAPEBRIDGE BULKREQUESTMIGRATIONMAXFILES –The maximum number of files in a migration set TAPEBRIDGE BULKREQUESTRECALLMAXBYTES –The maximum number of bytes a set of files to recall should represent TAPEBRIDGE BULKREQUESTRECALLMAXFILES –The maximum number of files in a recall set Developments for tape - 9

10. Data & Storage Services UNKNOWN tape drives fixed Drives stuck in UNKNOWN in the VDQM Operator had to manually DOWN and UP drives After 5 years bug was tracked to the new VDQM Race condition between an unused monitoring message and a state change message Race condition removed in 2.1.13-6 tape servers Developments for tape - 10

11. Data & Storage Services Architecture of mhvtl Developments for tape - 11 st driver sg driver SCSI mid-layer mhvtl driver (pseudo host bus adapter - HBA) vtltape daemon vtltape daemon vtllibrary daemon vtllibrary daemon rtcpd daemon rtcpd daemon rmcd daemon rmcd daemon User Space Kernel space

12. Data & Storage Services mhvtl configuration and data files Configuration files are in /etc/mhvtl –device.conf Specifies the tape library and drive devices –mhvtl.conf Default media capacity syslog verbosity of the vtltape and vtllibrary daemons Kernel module debugging level Data files are in /opt/mhvtl –One directory per tape Developments for tape - 12

13. Data & Storage Services rpms of mhvtl Available from the Enterprse Linux Repository SLC5 rpms –mhvtl-utils-1.4.4-1.el5.elrepo User-space daemons and clients –kmod-mhvtl-1.4.4-1.el5.elrepo Kernel module SLC6 rpms –mhvtl-utils-1.4.4-1.el6.elrepo.x86_64.rpm –kmod-mhvtl-1.4.4-1.el6.elrepo.x86_64.rpm Developments for tape - 13

14. Data & Storage Services mhvtl compression statistics Victor Kotlyar helped fix the mhvtl compression statics LOG SELECT: Fix off-by-one issues –https://github.com/markh794/mhvtl/commit/07d9bc4852da0d59fcdec70c3168d3e96f9226a8https://github.com/markh794/mhvtl/commit/07d9bc4852da0d59fcdec70c3168d3e96f9226a8 Implement Parameter Code Reset in LOG SELECT –https://github.com/markh794/mhvtl/commit/335785b493430323e4dbaacc0c89fe78b425a5dbhttps://github.com/markh794/mhvtl/commit/335785b493430323e4dbaacc0c89fe78b425a5db mhvtl-1.4.4 was created by Mark Harvey (MH vtl). Thanks to contact withdag@wieers.com this new version was built in general ELRepo (Enterprse Linux Repository). Developments for tape - 14