Write-through Cache System Policies discussion and A introduction to the system

Write-through Cache Disk File system Mount point: – /cache/halld (read-only cache system: /cache/mss/halld) Permission: – owned (writable) by a Unix group halld. Likes a normal file system, all members in the group can add/delete/modify the files. Disk to Tape library /mss mapping: – /cache/halld will map to /mss/halld – Volume set mapping is as same as /mss/halld – raw data volume set – still send CCPR if needs a new raw data directory

Write-through Disk Manager (WDM) Management: – Each group will have a quota and a reservation (quota is the maximum space a group can use and reservation is the minimum space guaranteed by cache manager). – Quota is soft number, when quota is exceed, user still can write to the file system and the older file will be delete in next clean cycle. Note: – Current jcache command will fail if the requested files are in the write-through disk pool.

WDM (Backup Policy) Backup Policy: – When a file is 12 days old, it will be backed up into tape library and considered as a read-only file. (to modify the file, user has to remove the copy in tape library first). – File with size lesser than 3 MB will not be backed up. Please tar the important small files so they can be backed up as one large file. User can backup a file anytime using utility ‘wcache put’. 12 days and 3MB is configurable parameters. – 12 days or other ?? – We like to set the min backup file size as larger as possible, what is the right size ??

WDM (Backup Policy) Since the new cache disk will map to old /mss/halld stub directory, the ‘wcache put’ will fail if user create a name conflicted file. CacheManager will marked this file in tape (doesn’t know it is different file). Only when cacheManager going to delete this file, it will discover it is a duplicated file. (see next slide)

WDM (Deletion Policy) Deletion Policy: – When disk space is needed, the oldest files that satisfied the criterion "pin count = 0 AND backed up" will be deleted. – Files less than 3 MB and is not accessed in 2 years will be deleted. After file is deleted from disk, it can be staged back to disk using utility ‘wcache get’. User can use utility ‘wcache delete’ to tell manager to delete file from disk if it is in tape library. Before each file is deleted, manager will make sure the copy on disk is same with the copy in tape library. (continues to next page)

Duplicated File - If copy on disk is different with the copy in tape library, a will be send to owner so you can delete the copy in tape or on disk. The copy on disk will be delete if no action is taken within 1 week.) The cache version of this file is different from the copy in the tape library. A proper action (remove the copy from tape or delete the cache copy) is needed. If no action is taken within 2 weeks, the copy on cache disk will be deleted. Please check this page on how to use 'wcache tapeRemove' tool to delete a file from Jlab tape library. /cache/junk/szscl21_xxxx_2316.limehttps://scicomp.jlab.org/docs/wcache We can change this behavior. Feedbacks are welcome!!!

WDM (Pin Policy) Any user in the group can pin files, but pin will fail if total pin in the group exceed 30% of total quota. Project manager can send a request to increase project quota if needed. File will not been pinned when ‘get file’ is called. User will pin file after it is on disk.

Best Practice When create new directories: – Set correct permission. – Use ‘correct name’, which means project related name, should not rename any top directory. When create new files: – Check files in /mss/halld (map to /cache/halld) When cache the files: -Cache the total data size. Some files maybe flushed out of disk when large amount (more than the quota) of cache is called.

WDM Utility - wcache wcache projectInfo projectName --- get project info specified by projectName (such as halld) wcache pin [-D days] life file1 file2... fileN --- pin specify file(s) for a given day (default 7 days) wcache unpin file1 file2... fileN --- unpin specify file(s) wcache pinInfo [-u user] [-n number] --- print user’s newest pin up to given number (default 100) wcache get file1 file2... fileN --- cache specify file(s) from tape library wcache put [–d] file1 file2... fileN --- backup file(s) into tape library and delete from disk if [-d] wcache requestStatus requestIndex --- get request status specified by request index wcache pendingRequest [-u user] ---- get unfinished request status wcache cancelRequest requestIndex ---- cancel a unfinished request specify by request index wcache checksum file --- print crc32 checksum of specify file wcache tapeRemove file1 file2... fileN --- remove specified file(s) from Jlab tape library wcache list [options] file1 file2... fileN --- list file properties (cacheManager related meta data) Note: Last two commands will coming soon.

WDM Utility - wcache Wcache client is installed at /site/bin/wcache. File path in argument must starts /cache/. Wild card in file path is not supported in this version. If there is any error detected before server performs the work, a text error message will print out. Error: Invalid file /cache/halld/bad_file (not in /mss) The arguments, options and return value may change during next few weeks. Please run ‘wcache –h’ for updated information.

wcache projectInfo /site/bin/wcache projectInfo halld projectName: halld rootPath: /cache/halld/ reserved: 1,000 GB quota: 10,000 GB availablePin: 3,221,225,472,000 cached: 176,426,721,312 pinned: 0 smallFile: 0 /site/bin/wcache projectInfo hall Error: Invalid project name 'hall'

wcache get wcache get /cache/junk/grid13.tar /cache/halld/good_file Error: no permission to create /cache/halld/good_file wcache get /cache/halld/bad_file /cache/junk/grid3.tar Error: Invalid file /cache/halld/bad_file (not in /mss) wcache get /cache/junk/123 /cache/junk/2316.lime get request: 23 status: pending /cache/junk/123 -> fail (not in tape library) /cache/junk/2316.lime -> pending

wcache get Since /cache/halld maps to /mss/halld. User can get any file under /mss/halld. But file path must starts with /cache/halld/… (not /mss/halld/…). ls /mss/halld/halld-scratch/hdops bigfile2 et2evio_ evio.001 et2evio_ evio.003 et2evio_ evio.005 et2evio_ evio.007 et2evio_ evio.009 et2evio_ evio.000 et2evio_ evio.002 et2evio_ evio.004 et2evio_ evio.006 et2evio_ evio.008 et2evio_ evio.011 Wcache get /cache/halld/halld-scratch/hdops/bigfile2 User must creates all parent directories before run ‘wcache get’. If /cache/halld/halld-scratch/hdops doesn’t exist, create all directories with correct group permission first.

wcache put wcache put /cache/junk/2316.lime /cache/junk/1234.lime Error: /cache/junk/1234.lime doesn't exist in cache disk wcache put /cache/junk/2324.lime put request: -1 status: done /cache/junk/2324.lime -> done (file already in mss) wcache put /cache/junk/2324.lime /cache/junk/2316.lime.save -d put request: 46 status: pending /cache/junk/2324.lime -> done (file already in mss) (will be deleted from disk soon) /cache/junk/2316.lime.save -> pending (will be deleted from disk after put finish)

Wcache pendingRequest /site/bin/wcache pendingRequest get request: 37 user: ychen status: pending /cache/junk/57262.A16 -> pending /cache/junk/57262.A16.f -> pending get request: 38 user: ychen status: pending /cache/junk/2316.lime -> pending put request: 41 user: ychen status: pending /cache/junk/2316.lime.save -> pending wcache pendingRequest -u chen chen has no pending request

wcache requestStatus /site/bin/wcache requestStatus 37 get request: 37 user: ychen status: pending /cache/junk/57262.A16 -> pending /cache/junk/57262.A16.f -> pending /site/bin/wcache requestStatus 230 Error: Invalid request index '230'

wcache pin wcache pin /cache/junk/2324.lime /cache/junk/2316.lime.save /cache/junk/2324.lime -> success (pinned for 7 days) /cache/junk/2316.lime.save -> success (pinned for 7 days) wcache pin -D 12 /cache/junk/2316.lime.save /cache/junk/2316.lime.save -> success (pinned for 12 days)

wcache pinStatus /site/bin/wcache pinInfo ychen pin 3 files and last 3 pins: /cache/junk/szsc_cfg_2324.lime pinned at :54:06.0 for 4 days /cache/junk/2316.lime pinned at :54:06.0 for 4 days /cache/junk/2324.lime pinned at :57:56.0 for 10 days /site/bin/wcache pinInfo -u chen chen has no pin.

wcache unpin wcache unpin /cache/junk/2324.lime /cache/junk/2316.lime /cache/junk/2324.lime -> success /cache/junk/2316.lime -> failed (not pinned)

Development Tasks Fix bugs and add new features (report problem and send suggestions) Better error handle ‘wcache tapeRemove’ will available soon. ‘wcache put -d’ will available soon. ‘wcache list’ will available soon. Wcache web display pages will available soon. Update Auger to handle ‘wcache get’/‘wcache pin’/‘wcache unpin’ for Input of from write-through cache disk if user think it is necessary. Admin page to create new project.

Changes in Batch Job – jsub script Specify a input file – Use syntax of files on /volatile – INPUT_FILES: /cache/halld/file.dat (not INPUT_FILES: /mss/cache/halld/file.dat ) – not Auger will not interact with wcache server to get/pin files under write-through disk pool /cache/halld. It is user’s responsibility to make sure the file is on cache disk

Changes in Batch Job – jsub script Output file – OUTPUT_DATA: file.out – OUTPUT_TEMPLATE: /cache/halld/outdir/file.out OUTPUT_TEMPLATE: /mss/cache/outdir/file.out – Auger will not jput the file to tape (just copy to cache disk) and cacheManager will do it after 12 days.

Changes in Batch Job – file stage Input file from /cache/halld/ will be copied to farm node. Output file to /cache/halld/ will be copy to cache disk (not tape library). At first Auger will not cache/pin/unpin for these files (assume the file is on disk during the life of farm job).