Re: why filling Hash seems slower on unixes than windows ?

You said: "the script in windows (along with some other pre and post processing) took about 25 seconds ..and on linux it took 135 seconds". Is this based off of the wall-clock? Or did you actually benchmark the code (use Benchmark;)? Wall-clock times are notorious for varying all over the map, depending on what else is running on the machine.

I had a 'Lead Programming Manager' once tell me that he'd "benchmarked some RPC code I wrote and it was giving him unacceptable response times" -- thirty seconds between the time he pressed the enter key and the time the browser got updated. After some investigation, I found the bug. It seems that there was a small error with his lap-top's DNS configuration. The first two DNS-servers in his authoritative list were non-existent, due to transposition errors, and the DNS time-out was set to 15 seconds. Once we fixed that, the response time he measured was sub-second. An actual benchmark gave 0.2 seconds from start to finish. (That was about his reaction time, pushing the start/stop timer button on his wrist-watch....)

When you start saying things about program timings, you have to be sure that you are really measuring the code's performance and not side-effects of the other things on the Machine.

----
I Go Back to Sleep, Now.

OGB

Comment on Re: why filling Hash seems slower on unixes than windows ? Download Code

Replies are listed 'Best First'.
Re^2: why filling Hash seems slower on unixes than windows ? by Fletch (Bishop) on Jan 03, 2008 at 18:41 UTC
And even once you're sure you're comparing at least apples to oranges (rather than apples to giant squid) you need to account for environmental differences. As was alluded to above, environment can greatly affect timing on multiuser boxen. If you're comparing benchmark times on your desktop Wintendo box which is running nothing else concurrently against benchmark times from a shared CGI server at your ISP chances are you're not going to get "meaningful" results from the comparison (at least not for meaningful values of "meaningful" :). The box could be oversubscribed and you're getting 75 wall-clock seconds of paging activity on one side; you could have a 10k RPM disk on your desktop while the other box is a 7 year old 5400 RPM SCSI drive that's the 6th device on a busy cable. Maybe the directory the source data resides in is being pulled over NFS from across the country. So no, it's not impossible to imagine plausible scenarios where the numbers you quote are the result. It is fairly impossible to say why though without more concrete details. But underneath it all there's nothing in perl itself per se that's going to make hashes slower or faster on any one implementation (again, it's how that implementation interacts with its environment that's going to introduce variances). Update: And just to expand on "meaningful results", if your target box takes 135 seconds then your target box takes 135 seconds; comparing against your desktop number doesn't make the other system run any faster. Doesn't really make much sense to get your desktop system's time down if the implementation you use misbehaves where you really need it to run (say slurping entire files into RAM which works fine until you toss it on the other box that starts to thrash if you load more than ~2M of data in any one process). Look at your algorithm, how your processing the data, and why that's taking as long as it is on your target box and fix that instead. The cake is a lie. The cake is a lie. The cake is a lie.	[reply]

Replies are listed 'Best First'.

Re^2: why filling Hash seems slower on unixes than windows ?
by Fletch (Bishop) on Jan 03, 2008 at 18:41 UTC

And even once you're sure you're comparing at least apples to oranges (rather than apples to giant squid) you need to account for environmental differences. As was alluded to above, environment can greatly affect timing on multiuser boxen.

If you're comparing benchmark times on your desktop Wintendo box which is running nothing else concurrently against benchmark times from a shared CGI server at your ISP chances are you're not going to get "meaningful" results from the comparison (at least not for meaningful values of "meaningful" :). The box could be oversubscribed and you're getting 75 wall-clock seconds of paging activity on one side; you could have a 10k RPM disk on your desktop while the other box is a 7 year old 5400 RPM SCSI drive that's the 6th device on a busy cable. Maybe the directory the source data resides in is being pulled over NFS from across the country.

So no, it's not impossible to imagine plausible scenarios where the numbers you quote are the result. It is fairly impossible to say why though without more concrete details. But underneath it all there's nothing in perl itself per se that's going to make hashes slower or faster on any one implementation (again, it's how that implementation interacts with its environment that's going to introduce variances).

Update: And just to expand on "meaningful results", if your target box takes 135 seconds then your target box takes 135 seconds; comparing against your desktop number doesn't make the other system run any faster. Doesn't really make much sense to get your desktop system's time down if the implementation you use misbehaves where you really need it to run (say slurping entire files into RAM which works fine until you toss it on the other box that starts to thrash if you load more than ~2M of data in any one process). Look at your algorithm, how your processing the data, and why that's taking as long as it is on your target box and fix that instead.

The cake is a lie.
The cake is a lie.
The cake is a lie.

[reply]