FastBit: interactively searching massive data
- 1 July 2009
- journal article
- Published by IOP Publishing in Journal of Physics: Conference Series
- Vol. 180 (1) , 012053
- https://doi.org/10.1088/1742-6596/180/1/012053
Abstract
As scientific instruments and computer simulations produce more and more data, the task of locating the essential information to gain insight becomes increasingly difficult. FastBit is an efficient software tool to address this challenge. In this article, we present a summary of the key underlying technologies, namely bitmap compression, encoding, and binning. Together these techniques enable FastBit to answer structured (SQL) queries orders of magnitude faster than popular database systems. To illustrate how FastBit is used in applications, we present three examples involving a high-energy physics experiment, a combustion simulation, and an accelerator simulation. In each case, FastBit significantly reduces the response time and enables interactive exploration on terabytes of data.Keywords
This publication has 22 references indexed in Scilit:
- Terascale direct numerical simulations of turbulent combustion using S3DComputational Science & Discovery, 2009
- Inverted files for text search enginesACM Computing Surveys, 2006
- The ROD crate DAQ software framework of the ATLAS data acquisition systemIEEE Transactions on Nuclear Science, 2006
- High-quality electron beams from a laser wakefield accelerator using plasma-channel guidingNature, 2004
- Bitmap index design and evaluationPublished by Association for Computing Machinery (ACM) ,1998
- The anatomy of a large-scale hypertextual Web search engineComputer Networks and ISDN Systems, 1998
- An overview of data warehousing and OLAP technologyACM SIGMOD Record, 1997
- The data warehouse and data miningCommunications of the ACM, 1996
- The STAR experiment at the relativistic heavy ion colliderNuclear Physics A, 1994
- An architecture for a business and information systemIBM Systems Journal, 1988