How would you use perl debugging tools to pinpoint the problem here?

tphyahoo has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: How would you use perl debugging tools to pinpoint the problem here? by BrowserUk (Patriarch) on Nov 10, 2005 at 15:35 UTC
First I would comment out half the top-level calls in the main program. In this case, commenting out `sub_2();` prevents the "crash". Having isolated (using a binary chop) which routine is causing the problem, I'd go into it, and assuming the problem didn't leap out at me as here, I'd comment out half of the body of the sub to find which half the problem lay in. That very quickly allows you to zero in on the line that is the problem. There are other ways, and some of them are definitely more applicable in other languages, but using a binary chop of comments works very well and quickly in Perl (and most dynamic langauges). Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re: How would you use perl debugging tools to pinpoint the problem here? by rinceWind (Monsignor) on Nov 10, 2005 at 15:34 UTC
`$hash->{$_}=1 for 1 .. 10000000;` [download] What you have here is known as a memory sponge. You don't need a debugger to see what is happening. You are asking the perl process to soak up (pretty well) as much RAM as you have on your system. Whether this will: Terminate with a message about exceeded resources (ulimit) Cause the machine to thrash and swap with its page file Lock up and require a reboot depends on which operating system you are running. Hope this helps -- Oh Lord, won’t you burn me a Knoppix CD ? My friends all rate Windows, I must disagree. Your powers of persuasion will set them all free, So oh Lord, won’t you burn me a Knoppix CD ? (Missquoting Janis Joplin)	[reply] [d/l]
Re: How would you use perl debugging tools to pinpoint the problem here? by tilly (Archbishop) on Nov 10, 2005 at 15:58 UTC
With a bit of proactive work you don't have to worry about crashing your computer. `use BSD::Resource; my $limit = 10010241024; # 100 MB setrlimit(RLIMIT_AS, $limit, $limit);` [download] If you're using Apache, you can do this by making sure that the setrlimit utility is in your PATH, and setting the environment variable APACHE_ADDRESS_SPACE_MB to whatever you want before starting Apache. This doesn't stop you from having to debug the problem, but this makes the bug be less of a development problem for you.	[reply] [d/l]
Re^2: How would you use perl debugging tools to pinpoint the problem here? by blazar (Canon) on Nov 10, 2005 at 16:38 UTC
`my $limit = 10010241024; # 100 MB` [download] Also, `my $limit = 100<<20; # 100 MB` [download] which I find to be better readable, but as usual YMMV.	[reply] [d/l] [select]
Re: How would you use perl debugging tools to pinpoint the problem here? by blazar (Canon) on Nov 10, 2005 at 16:33 UTC
The program can be even simpler: `#!/usr/bin/perl use strict; use warnings; my %hash; $hash{$_}=1 for 1 .. 10_000_000; __END__` [download] Your use of a hashref makes it slower because perl has to continuosly dereference it. But there's no need of any particular debugging tool to understand that the problem hast to do with your attempt to fill a hash with a huge number of keys. The fact is that perl runs quick out of memory and starts swapping to disk. At least this is what I can infer from the music my case just started playing... The fact that you're filling it an item at a time doesn't help either, but the keys are so many that even preassigning to keys won't help: `#!/usr/bin/perl use strict; use warnings; use constant KEYS => 10_000_000; my %hash; keys %hash=KEYS; $hash{$_}=1 for 1 .. KEYS; __END__` [download] You should really consider a db based approach or a tied hash solution instead. Also, I'm not really sure, and I may be utterly wrong, but I suspect that due to the nature of the keys, i.e. numbers, they may behave poorly to the hashing function. Incidentally, were they not that many, I'd initialize it like thus instead: `#!/usr/bin/perl use strict; use warnings; use constant KEYS => 10_000_000; my %hash; keys %hash=KEYS; @hash{1..KEYS}=(1) x KEYS; __END__` [download] (To be sure: this has the same problems as the above!)	[reply] [d/l] [select]
Re: How would you use perl debugging tools to pinpoint the problem here? by ikegami (Patriarch) on Nov 10, 2005 at 15:14 UTC
I'm curious... How do you define crash in this instance?	[reply]
Re^2: How would you use perl debugging tools to pinpoint the problem here? by tphyahoo (Vicar) on Nov 10, 2005 at 15:51 UTC
Well I'm on windows, and everything just gets slower and slower until it completely freezes up. ctrl-alt-delete kill to task manager and then kill process works to get things moving again.	[reply]
Re^3: How would you use perl debugging tools to pinpoint the problem here? by ikegami (Patriarch) on Nov 10, 2005 at 16:17 UTC
It's not crashing at all. A hash with 10 million elements is huge! Allocating a hash of that size -- at least 100MB, I bet -- takes time, and if the OS starts using virtual memory, it'll take forever.	[reply]
Re: How would you use perl debugging tools to pinpoint the problem here? by VSarkiss (Monsignor) on Nov 10, 2005 at 16:09 UTC
I ran into this problem "in the wild" and asked about it here. The really bad part is that you can't even catch it with `perl -c`. Do not rebuke them with harsh words ... but rather lead them gently - with URLs - so that they may learn wisdom.	[reply] [d/l]
Re^2: How would you use perl debugging tools to pinpoint the problem here? by blazar (Canon) on Nov 10, 2005 at 16:42 UTC
Personally I don't say any reason why it should be caught with `perl -c`. It's not a syntax error after all. To be precise it's not an error at all. Depending on the hardware, the OS and the OS configuration, it may run just smooth...	[reply] [d/l]
Re^3: How would you use perl debugging tools to pinpoint the problem here? by VSarkiss (Monsignor) on Nov 10, 2005 at 17:26 UTC
No, it's not an error, but it's a construct that, because of the well-meaning but misguided "optimization", causes the compiler to freeze up, and the OP was asking for ways that Perl itself, such as in the debugger, could help you figure out what was going on. Usually, `perl -c` is one of those ways, but not in this case. Brother tye's reply in that thread is an excellent illustration of what I think the problem is. Do not rebuke them with harsh words ... but rather lead them gently - with URLs - so that they may learn wisdom.	[reply] [d/l]


good chemistry is complicated, and a little bit messy -LW
	PerlMonks