Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: Using indexing for faster lookup in large file

by BrowserUk (Patriarch)
on Feb 27, 2015 at 20:15 UTC ( [id://1118123]=note: print w/replies, xml ) Need Help??


in reply to Using indexing for faster lookup in large file

You may find the subthread staring at Re: Index a file with pack for fast access of interest.

The indexing mechanism discussed there isn't directly applicable to your requirements -- it indexes by line number rather than content -- but it should be adaptable to them.

Assuming your sample data is representative -- ie. average record length 47 -- then 30GB represents ~700 million records. And assuming the key numbers are representative, you'd need a 32-bit int to represent those and a 64-bit int to represent the file offset. Hence your index could be built using:

open IN, '<', '/path/to/the/Datafile.txt' or die $!; open out, '>:raw', /path/to/the/indexfile.idx' or die $!; my $pos = 0; print( OUT pack 'VQ', m[^(\d+),], $pos ), $pos = tell( IN ) while <IN> +; close OUT; close IN;

The output file, with 12 bytes per record would be ~7.6GB.

As the keys in your file appear to be out of order, you would then need to (binary) sort that file.

Once sorted, a binary search would take an average of 30 seeks&reads to locate the appropriate 12-byte index record and another seek&read to get the data record.

If you have a sufficiently well-spec'd machine with (say) 8GB or more of ram, you could load the entire index into memory -- as a single big string and access as a ramfile -- which would probably reduce your lookup time by much more than an order of magnitude.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
; close OUT; close IN;

Replies are listed 'Best First'.
Re^2: Using indexing for faster lookup in large file
by anli_ (Novice) on Feb 27, 2015 at 23:11 UTC
    Hi, and thanks for your reply.

    The data isn't quite representable, and that was perhaps a bit stupid of me.A more proper representation is below (3 random lines).
    The data is sorted lexicographically on the first number.
    106896752;384407;root;cellular organisms;Eukaryota;Viridiplantae;Strep +tophyta;Streptophytina;Embryophyta;Tracheophyta;Euphyllophyta;Spermat +ophyta;Magnoliophyta;Mesangiospermae;eudicotyledons;Gunneridae;Pentap +etalae;rosids;fabids;Fabales;Fabaceae;Papilionoideae;Genisteae;Lupinu +s;Lupinus magnistipulatus; 124405058;5888;root;cellular organisms;Eukaryota;Alveolata;Ciliophora; +Intramacronucleata;Oligohymenophorea;Peniculida;Parameciidae;Parameci +um;Paramecium tetraurelia; 134053560;349161;root;cellular organisms;Bacteria;Firmicutes;Clostridi +a;Clostridiales;Peptococcaceae;Desulfotomaculum;Desulfotomaculum redu +cens;Desulfotomaculum reducens MI-1;
    In total there is about 160 million records
      The data is sorted lexicographically on the first number.

      You mean like this?

      C:\Users\HomeAdmin>perl -E"@s = 1..30; say for sort @s" 1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 23 24 25 26 27 28 29 3 30 4 5 6 7 8 9

      Also, what are the smallest and largest keys ("first numbers") in the file?


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1118123]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having an uproarious good time at the Monastery: (6)
As of 2024-04-19 07:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found