in reply to Memory Efficient Alternatives to Hash of Array

well, i don't know if you are familiar with SQLite (or MySQL), i often get stuck with processing large amount of data, at first i also tried to do it through hashes and arrays, but then i just gave up on it. my advise would be to use one of the freely available databases like SQLite or MySQL. just import data, sort it (order by), and when processing it retrieve line by line of data and do what ever you want to do with it.

this is memory efficient, but certainly slower way to do it

SQLite

example on how to use it through DBI and it would be usfull to check out DBD::SQLite

plus there are tons of concrete examples on perlmonks !!!

  • Comment on Re: Memory Efficient Alternatives to Hash of Array

Replies are listed 'Best First'.
Re^2: Memory Efficient Alternatives to Hash of Array
by tilly (Archbishop) on Dec 27, 2008 at 14:49 UTC
    Databases are effective in this case because they understand the importance of streaming data to/from disk rather than using it as RAM. As for why, see Re^2: Memory Efficient Alternatives to Hash of Array. However it is fairly easy for you to acquire the same knowledge, which basically comes down to knowing to sort then process. Armed with that knowledge, the database slows me down slightly for simple tasks. For complex tasks, the database saves some logic but may come up with an unworkable query plan. I'll therefore give it a try, but push comes to shove I'm willing to go out of the database because I know that I can make it work, and sometimes the database simply won't.
[reply]

In Section Seekers of Perl Wisdom