Script halting without error?

Jman1 has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks, Edit: Here's some code which causes the same error:

use strict;
use warnings;

my %hash = ();
foreach my $a (1..500) {
    foreach my $b (1..4000) {
        foreach my $c (1..10) {
            foreach my $d (1..7) {
                $hash{$a}{$b}{$c}{$d} = $d;
            }
        }
    }
}
print "here\n";
[download]

I've got a pretty big, oo perl script that's behaving strangely. The part in question essentially compares two graphs for similarity. For small datasets (~20-30 nodes per graph), it behaves correctly, but for large ones (200-300 nodes per graph) it just stops at some (incorrect) point with no error messages or warnings.

Some clues:
1) For the large dataset, perl's memory usage seems to climb a bit above 500 MB and then at some point jumps back down to around 30 MB.
2) Using Devel::Profiler works fine for the small sets, but for the large sets it tells me that there are some unstacked calls (which makes sense, since it seems to be halting in the middle.)

Any ideas?

Comment on Script halting without error? Download Code

Replies are listed 'Best First'.
Re: Script halting without error? by ambrus (Abbot) on Dec 12, 2005 at 19:38 UTC
I wonder if perl might have died from a fatal signal (like SIGSEGV), but your shell somehow doesn't report this? You can find this out by echoing the `$?` shell variable immediately after running the program from the shell. (This is for unix of course.) You mention that the script uses much memory. If the system runs out of memory, you may get a fatal signal instead of a proper error message from perl. Also, you can get a signal if the reference counting MM deallocates a long list: an example is `perl -we '$x = [$x] for 0 .. 100_000; warn 1; $x = (); warn 2;'` but you may have to adjust the number `100_000` for your system. If that's not the problem, then I guess you don't give enough information about the issue. You could try posting the script or even a simplified (non-functional) script that produces the same error.	[reply] [d/l] [select]
Re^2: Script halting without error? by Jman1 (Acolyte) on Dec 12, 2005 at 19:49 UTC
Sorry, I should have mentioned that I'm running on Windows using ActivePerl. A little more detail about the part that's dying: For each node n in graph 1 and m in graph 2 do a bunch of comparisons between n and m and come up with a score. Store this in a 3 dim hash: score{comparison}{n}{m}	[reply]
Re: Script halting without error? by spiritway (Vicar) on Dec 13, 2005 at 06:05 UTC
I'm wondering whether your script is dying, or just taking a very long time and seems to be hung. Does it actually exit? Is there any way you could boil your code down to a manageable length, to isolate the problem, and maybe post it here?	[reply]
Re^2: Script halting without error? by Jman1 (Acolyte) on Dec 13, 2005 at 06:30 UTC
It actually exits. I'll see what I can do to post a reproduction. Edit: Okay, this code does the same thing. (It stops and the print never shows up.) `use strict; use warnings; my %hash = (); foreach my $a (1..500) { foreach my $b (1..4000) { foreach my $c (1..10) { foreach my $d (1..7) { $hash{$a}{$b}{$c}{$d} = $d; } } } } print "here\n";` [download] I assume it's just running out of memory somehow, but I don't know why there's no error message or what I should do about it.	[reply] [d/l]
Re^3: Script halting without error? by zentara (Cardinal) on Dec 13, 2005 at 13:55 UTC
You didn't say what version of windows you are testing this on. I usually avoid windows, but I ran into a similar problem awhile back, when I was trying to see if a threaded program(which ran fine on linux) would work on windows. The program created a big hash of shared variables for the threads. On WindowsME, it just crashed after a bit. On WindowsXP it ran fine. So I know there is some big differences in the way Windows handles memory, between versions. The code was something like this: #############shared hashes for xml processor################# my @chs = (1..99); # setting this to 2 works on windowsME # but will bog down over 3, and crash # at values like 60 #WindowsXP ran it fine my $max_prog_chan = 60; #XP ok my %days; foreach my $channel(@chs){ foreach my $count(0..$max_prog_chan){ #print "$channel $count\n"; share $days{$channel}{$count}{'channel'}; share $days{$channel}{$count}{'channel_info'}; share $days{$channel}{$count}{'episode_num'}; share $days{$channel}{$count}{'start'}; share $days{$channel}{$count}{'stop'}; share $days{$channel}{$count}{'makedate'}; share $days{$channel}{$count}{'description'}; share $days{$channel}{$count}{'title'}; share $days{$channel}{$count}{'writer'}; share $days{$channel}{$count}{'director'}; share $days{$channel}{$count}{'actors'}; share $days{$channel}{$count}{'rating'}; share $days{$channel}{$count}{'length'}; share $days{$channel}{$count}{'category'}; share $days{$channel}{$count}{'star_rating'}; } } [download] I'm not really a human, but I play one on earth. flash japh	[reply] [d/l]
Re^3: Script halting without error? by spiritway (Vicar) on Dec 14, 2005 at 05:23 UTC
Well, you have 140 million iterations of an operation, or a set of operations. My question is, how do you know whether it's still grinding along, and hasn't finished, versus it chokes and dies. You say it stops and the print never shows up. Well, does the program end, and you get your cursor back? My guess is that your program doesn't stop, it just keeps struggling to finish the loops, and you have nothing to tell you it's still working. You might want to try adding a line in one of the earlier loops that prints something (even a period). `use strict; use warnings; my %hash = (); foreach my $a (1..500) { print "."; # <- Show we're alive by printing a dot... foreach my $b (1..4000) { foreach my $c (1..10) { foreach my $d (1..7) { $hash{$a}{$b}{$c}{$d} = $d; } } } } print "here\n";` [download] Anyway, that's just my thoughts on it - YMMV.	[reply] [d/l]