Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

You may find the subthread staring at Re: Index a file with pack for fast access of interest.

The indexing mechanism discussed there isn't directly applicable to your requirements -- it indexes by line number rather than content -- but it should be adaptable to them.

Assuming your sample data is representative -- ie. average record length 47 -- then 30GB represents ~700 million records. And assuming the key numbers are representative, you'd need a 32-bit int to represent those and a 64-bit int to represent the file offset. Hence your index could be built using:

open IN, '<', '/path/to/the/Datafile.txt' or die $!; open out, '>:raw', /path/to/the/indexfile.idx' or die $!; my $pos = 0; print( OUT pack 'VQ', m[^(\d+),], $pos ), $pos = tell( IN ) while <IN> +; close OUT; close IN;

The output file, with 12 bytes per record would be ~7.6GB.

As the keys in your file appear to be out of order, you would then need to (binary) sort that file.

Once sorted, a binary search would take an average of 30 seeks&reads to locate the appropriate 12-byte index record and another seek&read to get the data record.

If you have a sufficiently well-spec'd machine with (say) 8GB or more of ram, you could load the entire index into memory -- as a single big string and access as a ramfile -- which would probably reduce your lookup time by much more than an order of magnitude.


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
; close OUT; close IN;

In reply to Re: Using indexing for faster lookup in large file by BrowserUk
in thread Using indexing for faster lookup in large file by anli_

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (3)
As of 2024-04-25 17:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found