But I'm thinking that the first snippet of code walks through the data and gets the position, in bytes, for each starting number?

Correct. (But that "snippet" is a complete working program to create the index file. You did run that first didn't you?)

  1. It gets the position (in bytes) of the start of the record (using tell),
  2. and the first number on the line using the regex (m[^(\d+),]),
  3. and then packs the two into a 12-byte binary record and writes it to the index file.
Also running the code you wrote as-is only produced "Not found" for me.

The first thing that comes to mind is that you never answered my question above about the sort order of your data files.

If they are not sort numerically, then you will need to sort the file; or the index file; before the binary search will work.

It's seems as if the indexing might not catch what I would like since it won't find anything I'm looking for.

I'll need a little more information to go on.

Could you run the following steps in your console and copy&paste the output (in <code></code> tags).

Substitute whatever names you gave to the two programs above for 1118102-indexer.pl & 1118102-indexer.pl below.

The test file need only be a few dozen lines; but the lines must start with numbers, and it must be sorted numerically. The 200 sample lines you posted above would be ideal. :

>perl -V

>1118102-indexer smallFile.txt smallFile.idx

>1118102-searcher -N=10 smallFile.txt smallfile.idx

[download]

If you post the output from all 3 commands, it might give some clues as to what is going on.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

In reply to Re^3: Using indexing for faster lookup in large file (PP < 0.0005s/record) by BrowserUk
in thread Using indexing for faster lookup in large file by anli_

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.