in reply to File download statistics parsing
Pass 1: Build a hash using the path + size as the key and a count of the number of time you saw that combination as the value.
Pass 2: Iterate over the keys extracting the path and use stat or -s to get the real size. If when the size doesn't match, delete that entry from the hash.
Pass 3: Sum the sizes extracted from the remaining keys, * the associated counts. You have the total downloaded.
Build a hash containing path as the key and size as the value of all the files available for download. Save this in a file using Storable or similar.
Reload this hash at the start of the program and then as you parse the log, compare the size from the log from that in the hash and skip those where the size is disimilar.
The latter method is good if the list of files available is fixed or only changes infrequently. Just re-run the indexer script each time an new or modified file is made available. If the list is huge and the indexer takes a long time to run, have a third script that allows you to manually update the stroed hash with a new or modified entry.
|
|---|