Stegalex has asked for the wisdom of the Perl Monks concerning the following question:
I have a daily Perl script that runs and sends out several hundred emails (opt-in only of course). For the last several days, I have been noticing that it is not completing. It just "hangs" and stays on the process table. There is nothing printed either in the logs, in stderr, or in stdout. When I modified the program to print lots of debugging information, the output just stops halfway through printing a variable (it doesn't even complete printing the variable). My first thought was that I had run out of disk space, but running df showed that space was not a problem. Perhaps memory is a problem, although in the past, I have seen Perl print "out of memory!" to stderr and the program dies (not the program in question). This program is NOT dying, it just hangs out. It's not just moving slowly either and system performance does not seem to be impacted. I am sure that I am not trapped in an infinite loop because I have so many print statements that if I were in a loop, the logs would eventually consume all remaining disk space.
So, the question I have is this: If I keep the process running, is there a way to look inside to find out what's going on?
While I would love to post the code, company policy prohibits.
Thanks!
P.S. Perl 5.6.0 R.H. Linux Oracle 8i
Re (tilly) 1: Program hangs
by tilly (Archbishop) on Jan 27, 2002 at 23:22 UTC
|
If your output stops halfway through printing a variable,
the odds are that your debugging output is
Suffering From
Buffering. Fixing that won't fix your original problem,
but may help you debug it.
As for what it is, many possibilities exist. You might
be just triggering sendmail's check for being used to send
spam (at which point it deliberately slows down). Perhaps
you have one particular place you send information which
hangs for reasons beyond your control. (In which case it
might make sense to use Parallel::ForkManager to
run several processes in parallel, so that one hangs and
the rest go to their destination.) It could be something
else.
For a random example of "something else", I am in the
process of diagnosing a sporadic hang at work which
I have tracked down to probably being delays in closing
a database connection. (It is now in the hands of some
network folks.) | [reply] [Watch: Dir/Any] |
|
Hmmm. I am pretty sure it's not an SMTP problem as the program does not directly interact with SMTP, rather, it sends a command to another server which then forms and sends a mail message. Also, I am using qmail instead of sendmail. Thanks for your advice. I will look into your other tips.
| [reply] [Watch: Dir/Any] |
Re: Program hangs
by Fletch (Bishop) on Jan 27, 2002 at 22:07 UTC
|
A tool like strace that can dump out what system
calls are being made can provide insight into strange hangs
like this. You may want to have the services of a seasoned
C programmer or sysadmin handy if you're not familiar with
it.
| [reply] [Watch: Dir/Any] |
Re: Program hangs
by beebware (Pilgrim) on Jan 27, 2002 at 22:30 UTC
|
Trying running the code with perl -wc programname.pl as this has caught many hangs that I've experienced (sometimes just a closing multi-line quote missing can hang the entire thing).
Use use strict; as well to double-check everything, and if need be add debug lines (printing to the console - not to a file: you'll have less to go wrong if it is displayed rather than written) after every statement.
Oh, since you're using Oracle (I'm guessing via DBI), ensure that all your modules are up-to-date and turn on debugging on the DBI module as that will help as well. | [reply] [Watch: Dir/Any] [d/l] [select] |
Re: Program hangs
by n3dst4 (Scribe) on Jan 27, 2002 at 22:28 UTC
|
If it's single-threaded and you're sending emails, maybe an SMTP connection is getting hung. If neither you nor the far end server times out, it would appear that your program had stopped. Alternatively, you might have several slow connections, and when you come to look at the progress, it just happens to be stuck on one of them. How long have you left it just sitting there? It might come back after a while. | [reply] [Watch: Dir/Any] |
|
It is single-threaded, although it doesn't interact directly with SMTP, it passes a command to a different machine. The other machine actually forms the mail message.
The program is stopping printing right in the middle of printing a variable. This is the sort of thing that happens when you run out of disk space, although I have plenty of space in the filesystem.
The process can hang for days without moving. It took me a couple of days before I noticed the problem because there were no obvious signs of errors such as stderr, stdout, logfile entries.
| [reply] [Watch: Dir/Any] |
|
| [reply] [Watch: Dir/Any] [d/l] |
Re: Program hangs
by xylus (Pilgrim) on Jan 27, 2002 at 21:47 UTC
|
While I'm sure most of us here would love to help, but without a more specific idea of what is going on, there isn't much help to offer except to point you to some resources such as: perldebug.
However, if you could be more specific (perhaps psuedo-code) we could offer a little more insight.
-xylus | [reply] [Watch: Dir/Any] |
Re: Program hangs
by atcroft (Abbot) on Jan 27, 2002 at 23:29 UTC
|
Thoughts off the top of my head: - time-outs (mentioned above)
- memory/space issues (mentioned above)
- program code issues(mentioned above
- unexpected data
The first three were mentioned above, and in better detail than I could do.
What do I mean by that last one? The possibility of invalid data (invalid character in or bad/unexpected email address/name/whatever). You might consider wrapping your sending code/call in an eval() statement then examining the values of the $!, $@, $?, and/or other variables (as appropriate to your code) and logging any content they have (along with the data you used on that pass).
I hope that helps, and hope what monks wiser than I will chime in, so we both may learn.
| [reply] [Watch: Dir/Any] |
|
Good thought, but the data is clean. That's one of the first things I checked.
| [reply] [Watch: Dir/Any] |
A reply falls below the community's threshold of quality. You may see it by logging in. |
|
|