in reply to storing a huge text file into a hash

a particularly large text file ... but the act of storing the data is taking an impossibly long time

How big (in lines) and how long?

  • Comment on Re: storing a huge text file into a hash

Replies are listed 'Best First'.
Re^2: storing a huge text file into a hash
by Angharad (Pilgrim) on Dec 07, 2010 at 18:34 UTC
    its 574MB in size with over 11000000 lines
[reply]

      You didn't say how long it is taking on your system? On mine, this one liner loads 11e6 lines into a hash in a < 100 seconds: 

      >perl -e"BEGIN{keys %h=2**23}" 
-nE"$h{$_->[0]}=$_->[1] for [split];print qq[\r$.]" junk.dat
11000000

      [download]

      But it does use over 2GB of RAM.

      If your system is taking substantially longer than that, it could be that you are moving in to swapping, which would slow things down a lot.

      If you are loading this hash frequently, then you'd probably be better to stick your data into a tied DB like SQLite.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom