Re^2: Saving an array to a disk file

Dear Tanktalus,

You have a great intuition! Actually this posting is a continuation of my earlier question:

Howto Avoid Memory Problem in List::MoreUtils.

My basic problem is "Out of memory" error during process of getting uniq array. Unfortunately, the suggestion by salva, at some point also giving me same memory problem. So following your comment:

If you run out of memory, this will still swap to disk, but that shouldn't be slower than your method.

Seems that your solution (last snippet) is the best I can get. Avoiding run out of memory problem yet still significantly as fast as List::MoreUtils::uniq ? Please correct me if I'm wrong..

Comment on Re^2: Saving an array to a disk file

Replies are listed 'Best First'.
Re^3: Saving an array to a disk file by Tanktalus (Canon) on May 26, 2006 at 12:57 UTC
If you're running out of memory, increase your swap size. If you're on a unix/linux type of box, check your ulimit. Set it to unlimited (you may need superuser authority to do this). The only reason I can think of for a sysadmin to legitimitely say no is that you're still in school, and this is a school assignment. In that case, I'd suggest asking your professor for direction. Otherwise, it's either your machine at home (where you should already have superuser access - use it), or it's at work (where this is a work requirement and if the sysadmin says "no" then you ask your manager for help in turning that "no" to a "yes"). (You might be able to tell that I don't suffer fool admins well.) As for as fast as List::MoreUtils::uniq - I didn't remember about that function. This solution probably won't be as fast as that one if your array is already sorted. If your array is not sorted, then this solution removes the duplicates before sorting meaning that you have less to sort - that should make it faster: O(MlogM) instead of O(NlogN) where M is the number of unique values while N is the total number of values. (It's actually closer to MlogM + N, but under normal order notation, the N is lower-order, and thus discarded.)	[reply]
Re^4: Saving an array to a disk file by dsheroh (Monsignor) on May 26, 2006 at 16:32 UTC
As a (potentially fool) admin... I can think of reasons to legitimately say "no" which are not (in any sane workplace) subject to managerial override. They mostly fall into the category of "You want me to let you suck up all the memory on $RANDOM_MULTIUSER_PRODUCTION_HOST and bring other, potentially business-critical, processes to their knees as they wait for the disk to finish thrashing? No!" But, then, in a sane workplace, this sort of development would most likely be taking place on the developer's private workstation, or at least on a dedicated shared development machine, so those sorts of reasons wouldn't apply.	[reply]