1. Parallel I/O for Distributed Applications (MPI-Conn-IO)
SPP Tools
Users of MPI-2 Parallel I/O and parallel HDF5 with distributed applications
Objectives
MPI-2 parallel I/O subsystem that functions across machines
Set of management tools for handling files spread across multiple machines
Impact and DoD Relevance
Interoperation between different vendors’ MPI-2 I/O implementations and optimmization of inter-machine file transfer will achieve high performance remote access to large-scale data sets from distributed applications.
PET Team: G. Fagg and S. Browne, University of Tennessee-Knoxville
Methodology: The MPI-Connect system allows intercommunication between different MPI implementations using normal MPI communication calls. The MPI-Conn-IO extension to MPI-Connect supports shared access to remote files by parallel and distributed applications. The MPI-Conn-IO extension can be used either with MPI-Connect, in which case MPI-2 I/O calls are intercepted through the MPI profiling interface and dealt with appropriately, or in a stand-alone manner by non-MPI applications.
Results: The MPI-Conn-IO client library provides collective calls for reading and writing remote files. Tasks may request either replicated copies or partitioned views of a file. The file transfer protocol attempts to use network bandwidth efficiently by means of blocking, caching, and prefetching. Issues needing further study (e.g., consistency and security requirements) have been identified.
Significance: Performance of istributed DoD applications that access large-scale data sets will benefit from the MPI-Conn-IO use of tuned vendor versions of MPI-2 IO on a single platform and of prefetching and caching for multi-platform file access. MPI-Conn-IO support for MPI-2 IO and parallel HDF5 will help ensure portability of distributed applications that use those interfaces.
References:
S. Browne and G. Fagg, “A Parallel I/O Subsystem for Distributed Applications”, University of Tennessee Computer Science Technical Report (in preparation, to be submitted for publication)