Optimized Random Access File Downloading

Abstract

Significant amount of publicly availiable information is files available for download via HTTP. To access it, users are forced to download the entire file, which is often an inefficient way to retrieve data. For example, when you need to use file data immediately or you need to access only a portion of the file, normal file downloading results in undesirable overhead and network resource consumption.

In this article a method for optimizing the on-demand download of a file is described. It is based on downloading of exactly those parts of the file that are currenly being read by a client software. To maintain a balance between the speed of execution of the next read request and the total level of associated overhead, an effective download method is proposed. It reduces the maximum waiting time of the read request execution of by an order of magnitude with an average overhead rate of 10%.

The mechanism of sequential read requests detection and predicting future calls to fragments of the file is described, which reduces the level of overhead costs by 2-10 times for partial file reading. To this end, a probabilistic model for determining the size of fragments which are most likely to be read sequentially is proposed. It is based on the two-parameter Weibull distribution and allows flexible configuration of its parameters for effective use with different access patterns.