1. NCAR storage accounting and analysis possibilities David L. Hart, Pam Gillman, Erich Thanhardt NCAR CISL July 22, 2013 dhart@ucar.edu

2. Outline Rationale Archive storage accounting Disk storage accounting Analysis/reporting examples –Storage growth over time –Top consumers –Systemic changes in aggregate behavior –Weekly activity and patterns –Compute v. storage 2

3. Why storage accounting? Big Data –Increasing cost of storage with respect to compute NSF data management plan mandate –Tools for users Some info is better than no info –Some process is better than ad hoc fire drills Supports allocation processes 3

4. Accounting for archive storage NCAR has “charged” users for archive use for many years. –Archive accounting has institutional inertia NCAR HPSS details, June-July 2013 4 Date Files (M) PB (unique) PB (2nd copy) UsersTB+ 6/2/13137.619.522.3991181 6/9/13138.219.822.6991307 6/16/13138.820.122.9992370 6/23/13141.120.523.3998347 6/30/13142.420.723.51002266 7/7/13142.520.923.61005135

5. Archive storage record Activity date – date record was collected Activity type – Read, Write, Storage Unix uid Project code – project to charge Number of files Bytes – read, written, or stored Class of service – e.g., single-copy, dual-copy DNS – of client host Frequency – interval, in days, between accounting runs 5

6. Collecting data from HPSS Read/write activity –Analyze logs from HSI and HTAR (since May 2013). Logs archived daily, processed weekly. Storage activity –Weekly DB2 table scan and separate post-processing steps. Accounting system impact –Approx. 6,000 records per week Major accounting requirements –Use of HPSS accounting hooks to associate NCAR project code with HPSS file “account” –Accounting system and HPSS enforce requirement for every user to have a “default project” to which files will be charged if no other project provided 6

7. Accounting for disk storage Focus on long-term project spaces, which are allocated –But mechanism captures scratch snapshots, too! GLADE total storage, June-July 2013 7 DateFiles (M)PBUsersTB+ 6/8/13183.052.872,50655.3 6/15/13192.962.972,52599.3 6/22/13210.323.022,49053.1 6/29/13212.803.112,50089.5 7/6/13224.763.112,5098.8

8. Disk storage record Event time – date record was collected Project directory Group — Unix group Username Number of files kB used Period — reporting interval, in days QOS — a quality of service field (for future use) 8

9. Collecting data from GPFS File systems don’t have concept of “project”, but GPFS has notion of “file sets” –Leverage file sets to map to project spaces –For scratch, work, home: report per-user data Process runs weekly, provides a storage snapshot –With GPFS tools, process requires only a few minutes to complete—full file system scan not required Accounting system impact –Approx. 4,000 records per week Major accounting requirements –Agreements and processes between GLADE administrators and User Services about how spaces are created –Deviation would break the system 9

10. ANALYSIS AND REPORTING 10

11. Storage growth over time (1) HPSS growth in 2013GLADE growth in 2013 11

12. Storage growth over time (3) 12 User reports show project by week and per- user breakdown

13. Top consumers 13

14. Aggregate behavior (1) 14 Net growth, 3/3-4/7 — ~261 TB

15. Aggregate behavior (2) 15 Data written, 3/3-4/7 — 594 TB

16. Compute v. storage (1) 16

17. Compute v. storage use (2) 17

18. Big compute != Big data 18

19. What is “Big Data”? 19 Average file size vs.Total data holdings

20. Managing “orphaned” files Verifying accounting records lets site operators identify files owned by inactive users or inactive projects On July 7, HPSS accounting showed 177 users with 885 TB of “orphaned” files Early outreach to users and project leads does translate to deletions and fewer files for whom an owner cannot be found –Users required to be “actively engaged” in the disposition of their archive holdings. www2.cisl.ucar.edu/docs/hpss/policies 20

21. QUESTIONS? 21