MTIO. A multi-threaded parallel I/O system
- 22 November 2002
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 368-373
- https://doi.org/10.1109/ipps.1997.580928
Abstract
Presents the design and evaluation of MTIO (Multi-Threaded Input/Output), a multi-threaded runtime library for parallel I/O. We extend the multi-threading concept to separate the computation and I/O tasks into two separate threads of control. Multi-threading in our design permits (a) asynchronous I/O even if the underlying file system does not support asynchronous I/O; (b) copy avoidance from the I/O thread to the compute thread by sharing address space; and (c) a capability to perform collective I/O asynchronously without blocking the compute threads. Further, this paper presents techniques for collective I/O which maximize load balance and concurrency while reducing communication overhead in an integrated fashion. Performance results on an IBM SP2 for various data distributions and access patterns are presented. The results show that there is a tradeoff between the amount of concurrency in I/O and the buffer size designated for I/O, and that there is an optimal buffer size beyond which the benefits of larger requests diminish due to large communication overheads.Keywords
This publication has 7 references indexed in Scilit:
- PIOUS: a scalable parallel I/O system for distributed computing environmentsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Physical schemas for large multidimensional arrays in scientific computing applicationsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Dynamic file-access characteristics of a production parallel scientific workloadPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- The Nexus Approach to Integrating Multithreading and CommunicationJournal of Parallel and Distributed Computing, 1996
- The galley parallel file systemPublished by Association for Computing Machinery (ACM) ,1996
- PPFSPublished by Association for Computing Machinery (ACM) ,1995
- An experimental performance evaluation of Touchstone Delta Concurrent File SystemPublished by Association for Computing Machinery (ACM) ,1993