Estimating frequency of change
Top Cited Papers
- 1 August 2003
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Internet Technology
- Vol. 3 (3) , 256-290
- https://doi.org/10.1145/857166.857170
Abstract
Many online data sources are updated autonomously and independently. In this article, we make the case for estimating the change frequency of data to improve Web crawlers, Web caches and to help data mining. We first identify various scenarios, where different applications have different requirements on the accuracy of the estimated frequency. Then we develop several "frequency estimators" for the identified scenarios, showing analytically and experimentally how precise they are. In many cases, our proposed estimators predict change frequencies much more accurately and improve the effectiveness of applications. For example, a Web crawler could achieve 35% improvement in "freshness" simply by adopting our proposed estimator.Keywords
This publication has 5 references indexed in Scilit:
- Keeping up with the changing WebComputer, 2000
- World Wide Web caching: the application-level view of the InternetIEEE Communications Magazine, 1997
- Bayesian TheoryPublished by Wiley ,1994
- Methods of Mathematical PhysicsPublished by Wiley ,1989
- A Bayesian Approach to Parameter and Reliability Estimation in the Poisson DistributionIEEE Transactions on Reliability, 1972