Re: storing a huge text file into a hash

Replies are listed 'Best First'.
Re^2: storing a huge text file into a hash by Angharad (Pilgrim) on Dec 07, 2010 at 18:34 UTC
its 574MB in size with over 11000000 lines	[reply]
Re^3: storing a huge text file into a hash by BrowserUk (Patriarch) on Dec 07, 2010 at 18:58 UTC
You didn't say how long it is taking on your system? On mine, this one liner loads 11e6 lines into a hash in a < 100 seconds: `>perl -e"BEGIN{keys %h=2**23}" -nE"$h{$_->[0]}=$_->[1] for [split];print qq[\r$.]" junk.dat 11000000` [download] But it does use over 2GB of RAM. If your system is taking substantially longer than that, it could be that you are moving in to swapping, which would slow things down a lot. If you are loading this hash frequently, then you'd probably be better to stick your data into a tied DB like SQLite. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]