Workflow task clustering for best effort systems with Pegasus
- 29 January 2008
- proceedings article
- Published by Association for Computing Machinery (ACM)
Abstract
Many scientific workflows are composed of fine computational granularity tasks, yet they are composed of thousands of them and are data intensive in nature, thus requiring resources such as the TeraGrid to execute efficiently. In order to improve the performance of such applications, we often employ task clustering techniques to increase the computational granularity of workflow tasks. The goal is to minimize the completion time of the workflow by reducing the impact of queue wait times. In this paper, we examine the performance impact of the clustering techniques using the Pegasus workflow management system. Experiments performed using an astronomy workflow on the NCSA TeraGrid cluster show that clustering can achieve a significant reduction in the workflow completion time (up to 97%).Keywords
Funding Information
- Office of Cyberinfrastructure (OCI-0722019)
This publication has 8 references indexed in Scilit:
- A provisioning model and its comparison with best-effort for performance-cost optimization in gridsPublished by Association for Computing Machinery (ACM) ,2007
- How are Real Grids Used? The Analysis of Four Grid Traces and Its ImplicationsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2006
- Overhead Analysis of Grid Workflow ApplicationsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2006
- Optimizing Grid-Based Workflow ExecutionJournal of Grid Computing, 2005
- Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed SystemsScientific Programming, 2005
- Using Moldability to Improve the Performance of Supercomputer JobsJournal of Parallel and Distributed Computing, 2002
- Performance-effective and low-complexity task scheduling for heterogeneous computingIEEE Transactions on Parallel and Distributed Systems, 2002
- Static scheduling algorithms for allocating directed task graphs to multiprocessorsACM Computing Surveys, 1999