in reply to Use temporary file or cache

I then loop through the xml file again to find and print out all the actual blog posts. On each pass I search the entire temporary file for any comments that start with the post url and add those comments to the post

...

Is there a more efficient way of doing this?

Yes. Store the the comments with the post URL as the key in an hash of arrays, ie 

my %comments = (
   'http://url.to/post' => [ $comment1, $comment2, .. ],
    ...
)

[download]

And on the second pass simply look it up.

If the data is too large and doesn't fit all into memory, use something like DBM::Deep to store it in a file, but preserve fast lookup.

Replies are listed 'Best First'.
Re^2: Use temporary file or cache
by andrewheiss (Initiate) on May 28, 2009 at 09:20 UTC
    How big is too large? My largest xml file is about 20 mbs... I'm a little unclear on how to programatically save everything to the hash and then look it up... I haven't worked with hashes that much in Perl yet. So, the first time I loop through the xml to find the comments, I store each comment as a new entry in the %comments array, right? Something like this? 
    ...
my $commentText = $date . $author . $content;
my %comments = (
  $posturl => [$commentText]
)
...

    [download]

    I can see that working for one comment--would a second comment then overwrite the first in the array or would it add automatically? Would I need to use push or something?

    Then, during the second pass, how exactly would I reference the keyed array?

    Sorry for all the newbie questions... Thanks!

[reply]
[d/l]
      Try to read perlintro or perldata for more information on hashes, and perlreftut for more involved data structures. Suppose you have a comment stored in $comment, and want to store that it's related to the url $post_url, you'd write: 
      my %comments;
...
push @{$comments{$post_url}}, $comment;

      [download]

      And you can retrieve and iterate over the list of comments to an URL: 

      for my $c (@{$comments{$post_url}}) {
    print "$c\n";
}

      [download]
[reply]
[d/l]
[select]

      20mbs, even with a few copies in hashes and arrays here and there, probably fits in memory without any problem. There is some overhead in perl variables and more in the more complex structures (hashes and arrays) but even so you should be OK unless you have very limited virtual memory available. I would keep everything in memory (from perl's perspective) and let the swapper deal with disk if necessary, unless this proved to be problematic. You might have a look to see how much free RAM is available on your system.

[reply]

In Section Seekers of Perl Wisdom