spx2:

Ummm .... the definition of big has changed over the years. The graph size you mention used to be big, but now it's really not all that large. Before adding overhead, the graph is just 600*172800 bytes, just over 100MB. Double that for overhead, and you're still well within the typical virtual memory size (normally 'bout 2GB on an x86).

So if you actually exceed the physical memory size of your computer, then the OS will handle the shuffling of virtual/physical pages for you. Your problem will have to be nearly 20x larger to worry about funky techniques. So don't worry about it until you actually run out of memory!

Now in the event that your edges are large and plentiful and you actually approach the virtual memory size of your machine, then you can use a file of fixed record sizes, and for the node data just store the node number (record number). You can cache 'n' of 'em in memory so you don't spend all your time on Disk I/O. (Generalizing to variable-record length records is a little harder, but basically the same.)

Using 'Tie' to connect your file to the nodes is probably a good idea, but I don't know having never used it before.

...roboticus

In reply to Re: storing huge graph by roboticus
in thread storing huge graph by spx2

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.