Resource-Aware Distributed Stream Management Using Dynamic Overlays
- 1 January 2005
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636927,p. 783-792
- https://doi.org/10.1109/icdcs.2005.69
Abstract
We consider distributed applications that continuously stream data across the network, where data needs to be aggregated and processed to produce a 'useful' stream of updates. Centralized approaches to performing data aggregation suffer from high communication overheads, lack of scalability, and unpredictably high processing workloads at central servers. This paper describes a scalable and efficient solution to distributed stream management based on (1) resource-awareness, which is middleware-level knowledge of underlying network and processing resources; (2) overlay-based in-network data aggregation; and (3) high-level programming constructs to describe data-flow graphs for composing useful streams. Technical contributions include a novel algorithm based on resource-aware network partitioning to support dynamic deployment of data-flow graph components across the network, where efficiency of the deployed overlay is maintained by making use of partition-level resource-awareness. Contributions also include efficient middleware-based support for component deployment, utilizing runtime code generation rather than interpretation techniques, thereby addressing both high performance and resource-constrained applications. Finally, simulation experiments and benchmarks attained with actual operational data corroborate this paper's claimsKeywords
This publication has 16 references indexed in Scilit:
- Flux: an adaptive partitioning operator for continuous query systemsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2004
- Hermes: a distributed event-based middleware architecturePublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- ACDS: Adapting computational data streams for high performancePublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- TAGPublished by Association for Computing Machinery (ACM) ,2002
- Monitoring Streams — A New Class of Data Management ApplicationsPublished by Elsevier ,2002
- Infopipes for composing distributed information flowsPublished by Association for Computing Machinery (ACM) ,2001
- ChordPublished by Association for Computing Machinery (ACM) ,2001
- The state of the art in distributed query processingACM Computing Surveys, 2000
- ARMADA Middleware and Communication ServicesReal-Time Systems, 1999
- Performance tradeoffs for client-server query processingACM SIGMOD Record, 1996