... I need a method of being able to navigate around the file in as effeicient and speedy a manner possible... Any code that can read the file also needs to be able to write to it so that the file may be amended - currently this gets done by hand...

And are you planning for the navigation to be done by hand as well (i.e. interactively: the user starts the program, then carries on some sort of dialog to get to a point of interest, make changes if needed, jump to another point, make changes, and so on, until a final save/exit)?

Given that you can't redesign the file structure, that the files get to be up to 2 GB, and that the navigation/updates are to be controlled manually, my next question would be: is this just a one-shot or occasional process, or is it rather something that will be a continuing need, such that some extra code development -- and some extra cpu cycles the first time a given file is processed -- is justified?

If you need to handle a lot of files this way, and especially if you need to revisit any given file numerous times, it will be worth your while to use a database (e.g. mysql, which is easy to install if you don't have it already -- and installing the Perl DBI and DBD::mysql shouldn't be much trouble either).

Whatever creates the files in the first place does not need to change, and whatever uses the files after your seek/edit process is done can likewise remain unchanged. All you need is a front-end process to load a file into the database, a user interface to navigate and update the database records for a given file, and a back-end process to dump the database contents to a new file. The front- and back-end processes will run pretty quickly, and the UI can be optimized in any number of ways to be very quick and reliable, depending on what sort of information is needed to navigate.

In reply to Re: How to get fast random access to a large file? by graff
in thread How to get fast random access to a large file? by gothic_mallard

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.