in reply to Re: Giant Tie'd data structures
in thread Giant Tie'd data structures
Thank you all for your kind words and your assistance. For more detail about what I'm trying to do;
I have a large dataset of related files, stored in what is essentially a flat file. There is a "source" file, and a "destination" file, and each pair is listed on a single line. There may be many destination files for a single source file. (Read: the same source file may be listed more than once, but the destination files are all unique.)
There is a characteristic of some of these files that is undesirable: it is Unicode, or is some other, similarly odd filetype about it. I can measure that separately. The fact that some of these files have an odd filetype taints all the destination files as well.
I am trying to generate a complete list of all the bad files, along with their related, destination files if the bad file is the source of one of these related pairs.
If I do two scans--one to build the list of bad files, and then one to build the list of related files, it will take the script 12 hours to run, but I will complete the operation successfully. If I can reduce the lengthy scans to just one (by building a tie'd hash or btree that I can practically instantly scan through) I can reduce that to 6.
Anyway, thanks to the hints here, I switched to a btree tie which does the job no problemo. I will probably attempt to switch back to a hash now that I've learned I was tweaking the wrong tunable, but as long as I have a reasonably successful result, I'm a happy camper.
Thank you all! :-)