Re^2: Netflix (or on handling large amounts of data efficiently in perl)

Thanks for your suggestions. Currently trying to push the data out into a BerkeleyDB now having spent a few hours this morning trying to get an understanding of bdb usage. Gave up trying to understand MLDMB & bdb for now, the documentation on CPAN just got a bit weird. Found great resources using DB_File but sadly ActivePerl haven't managed to get that into their repository so far (I really miss having a *nix box around when it comes to this stuff!)

Vec? Urgh, more time wading through perldoc ahead. Great technical resource, but half of it can be a pain for anyone not from a comp-sci or c++ programming background!

Comment on Re^2: Netflix (or on handling large amounts of data efficiently in perl)

Replies are listed 'Best First'.
Re^3: Netflix (or on handling large amounts of data efficiently in perl) by tilly (Archbishop) on Dec 24, 2008 at 23:24 UTC
Random tip. Try http://strawberryperl.com/ and see if it lessens the pain of Windows. A more technical tip. Try sorting your data and using a btree format for your data. With a hash you do a lot of seeking to disk, and seeks to disk are slow. 1/200th of a second per seek may not sound like a lot, but try doing 100 million of them and you will take the better part of a week. But a btree loaded and accessed in close to sorted order does lots of streaming to/from disk and that is quite fast. (And a merge sort streams data very well.)	[reply]

Replies are listed 'Best First'.

Re^3: Netflix (or on handling large amounts of data efficiently in perl)
by tilly (Archbishop) on Dec 24, 2008 at 23:24 UTC

http://strawberryperl.com/

A more technical tip. Try sorting your data and using a btree format for your data. With a hash you do a lot of seeking to disk, and seeks to disk are slow. 1/200th of a second per seek may not sound like a lot, but try doing 100 million of them and you will take the better part of a week. But a btree loaded and accessed in close to sorted order does lots of streaming to/from disk and that is quite fast. (And a merge sort streams data very well.)

[reply]