Distributed computing with load-managed active storage
- 25 June 2003
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
Abstract
One approach to high-performance processing of massive data sets is to incorporate computation into storage systems. Previous work has shown that this active storage model is effective for a variety of problems. This paper explores opportunities to use active storage as a basis for exploiting asymmetric parallelism in applications using a streaming computation model on collections of fixed-size records. This model is the basis for much of the research in I/O-efficient algorithms, which deals with an important class of massive data problems not studied in previous work on active storage. We present an extension of a streaming computation model for an external memory toolkit to support a flexible mapping of computations to storage-based processors. Our approach enables load-managed active storage: it exposes parallelism, ordering constraints, and primitive computation units to the system, which can configure the application to balance load and make the best use of available processing power Emulation results from a sorting application demonstrate the potential of dynamic adaptation in load-managed active storage.Keywords
This publication has 18 references indexed in Scilit:
- The architectural costs of streaming I/O: A comparison of workstations, clusters, and SMPsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Optimizing execution of component-based applications using group instancesPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Highly concurrent shared storagePublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Flow computation on massive gridsPublished by Association for Computing Machinery (ACM) ,2001
- SEDAPublished by Association for Computing Machinery (ACM) ,2001
- External memory algorithms and data structuresACM Computing Surveys, 2001
- The Ninja architecture for robust Internet-scale systems and servicesComputer Networks, 2001
- Designing and mining multi-terabyte astronomy archivesPublished by Association for Computing Machinery (ACM) ,2000
- A cost-effective, high-bandwidth storage architecturePublished by Association for Computing Machinery (ACM) ,1998
- High-performance sorting on networks of workstationsPublished by Association for Computing Machinery (ACM) ,1997