Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re^2: storing a huge text file into a hash

by Angharad (Pilgrim)
on Dec 07, 2010 at 18:34 UTC ( #875856=note: print w/replies, xml ) Need Help??


in reply to Re: storing a huge text file into a hash
in thread storing a huge text file into a hash

its 574MB in size with over 11000000 lines
  • Comment on Re^2: storing a huge text file into a hash

Replies are listed 'Best First'.
Re^3: storing a huge text file into a hash
by BrowserUk (Patriarch) on Dec 07, 2010 at 18:58 UTC

    You didn't say how long it is taking on your system? On mine, this one liner loads 11e6 lines into a hash in a < 100 seconds:

    >perl -e"BEGIN{keys %h=2**23}" -nE"$h{$_->[0]}=$_->[1] for [split];print qq[\r$.]" junk.dat 11000000

    But it does use over 2GB of RAM.

    If your system is taking substantially longer than that, it could be that you are moving in to swapping, which would slow things down a lot.

    If you are loading this hash frequently, then you'd probably be better to stick your data into a tied DB like SQLite.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://875856]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (7)
As of 2023-02-05 14:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    I prefer not to run the latest version of Perl because:







    Results (31 votes). Check out past polls.

    Notices?