1. SDM Center Coupling Parallel IO with Remote Data Access Ekow Otoo, Arie Shoshani, Doron Rotem, and Alex Sim Lawrence Berkeley National Lab.

2. SDM Center SciDAC All Hands Meeting October’052 Outline  Project objectives  Status and accomplishments  Usage in an application  Extensions  Other future work

3. SDM Center SciDAC All Hands Meeting October’053 Project Objectives Development of the MpiioSrm library  mpiiosrm.h libmpiiosrm.a, libmpiiosrm.so Allows near-online access to files on mass storage system (e.g., HPSS), from MPI applications on a linux cluster Access files from local and remote MSS with MPI applications. Applications on a Linux cluster having  Local parallel file system (PVFS2) and  HPSS as the remote mass storage system

4. SDM Center SciDAC All Hands Meeting October’054 Status – 1 Libmpiiosrm Module Dependencies MPI Applications MPI-IOSRM PVFS2GPFSUFSXFSOther BDBOther ADIO HPSS SAM Jasmin Castor High Level Access and Control Record Structured File Access Low Level File System Access pNetCDF HDF5 mpiiosrm

5. SDM Center SciDAC All Hands Meeting October’055 Status – 2 Main Functions Functions in libmpiiosrm.a: 1.MPI_File_srm_proxy_init(); 2.MPI_File_srm_open(); [in place of MPI_File_open()] 3.MPI_File_srm_close(); [in place of MPI_File_close()] 4.MPI_File_srm_delete(); [in place of MPI_File_delete()] 5.MPI_File_srm_proxy_destroy() Functions (2) and (4) take a file name as one of its parameters. Note name changes from last meeting.

6. SDM Center SciDAC All Hands Meeting October’056 Status – 3 Major Changes Since Last AHM Function names revised MPI_File_srm_proxy_init() function starts an SRM client as a detached thread.  Only the process with the proxy_rank spawns this thread. Use of PVFS2 srm_put() implemented for MPI_File_writes, i.e.,  Files can now migrate from PVFS2 to HPSS.  Still being tested.

7. SDM Center SciDAC All Hands Meeting October’057 Usage in an Application Steps for reading remote files:  Prepare an input file for the program A file containing the file names to be read from HPSS if not found in local parallel file system.  Initiate grid-proxy-init(); password, etc. User requires a grid certificate  Start a namesrver, drmServer and a trmServer  Compile the program to be executed  Run “mpiexec –n XX ” to access files given in the input file.

8. SDM Center SciDAC All Hands Meeting October’058 Usage in an Application Input file Layout Implicit layout of parallel files  Uses default PVFS configuration  Alternatively use keys of MPI-IO File hints  Specify only pairs of source and destination URLs Explicit layout specifies for each file:  Pairs of source and destination URL  Start_IO_Node  Striping factor  Striping unit

9. SDM Center SciDAC All Hands Meeting October’059 Usage in an Application Program Skeleton … MPI_Init(); … MPI_Info_create(); MPI_Info_set(); … MPI_File_srm_proxy_init(); … MPI_File_srm_open(); … … MPI_File_srm_close(); MPI_File_srm_proxy_destroy(); … MPI_Finalize();

10. SDM Center SciDAC All Hands Meeting October’0510 Extensions MPI Applications Srm- -Client HPSS SAM Jasmin Castor DRMTRM Srm-Server MPI-IO File Control Data vs File Access Multi-site Access Fault tolerance & Failsafe

11. SDM Center SciDAC All Hands Meeting October’0511 Extensions - 2 Control of Prefetching of File Bundles  Process files, one at a time, by availability  Process files, one at a time, by sequence  Process files by bundles Data Access instead of File Access only  Allow for file filtering at the source SRM  Use of select criteria and indexes to generate only relevant data

12. SDM Center SciDAC All Hands Meeting October’0512 Extension - 3 Multi-Site Access  Extend access to other MSS implementing SRM specs.  Access files from multiple sites in a session  Extensions to Xrootd servers Fault Tolerance and Failsafe Operations  Easier now with multiple srm_client proxies being spawn as threads Access from C++ and Fortran

13. SDM Center SciDAC All Hands Meeting October’0513 Other Future Work Parallel Multidimensional Index Schemes  Repertoire of high and low dimensional indexing methods for scientific applications High dimensions:  Bitmaps (John, Kurt, etc)  Others Low Dimensions (1 ~ 8)  R-Tree, Order Preserving Extendible hashing,  Multi-level Grid File  String Searching Methods – Suffix trees, PATRICIA, etc.

14. SDM Center SciDAC All Hands Meeting October’0514 Other Future Work - cont. Extendible Multidimensional Array Files  With extendibility in all dimensions, not just one  For both dense arrays and sparse arrays  Efficiently accessible in MPI with irregular distributed array method using map arrays.  Multi-resolution array files

15. SDM Center SciDAC All Hands Meeting October’0515 Other Proposed Activities cont. Array Mapping Method for k dims, A[N]…[N] Element Access Ops with E extensions and constant k. Storage Size, element size s, integer size w Conventional Method, Extendible in 1 dimension only O(1)w*k + s*N k Index Array, Extendible in any dimensions O(1)w*N*k*(k+1) + s*N k Index Array Tree, Extendible in any dimensions O(ln E)w*((k+6)*E - 3) + s*N k

16. SDM Center SciDAC All Hands Meeting October’0516 Example of Mapping Function 102 345 678 91011 12 14 13 15 16 18 17 19 21202223 25 24 262728 30 29 3231 33 34 35 37 36 39 38 41 40 43 42 44 45 46 47 49 48 51 50 53 52 55 54 565758596061 64 63 65 62 676669687071 72 74 73 76 75 77 78 79 82 81 84 83 85 87 86 88 89 0 1 2 0 1435 2 3 4 7689 5 6 7 8 i0i0 i1i1 01 4 1300 7 4 1 204 87156 5 3 8 1 6 < 35 5 < < 17298 >= Red-Black-Like Binary Tree

17. SDM Center SciDAC All Hands Meeting October’0517 The End