Thanks for your help
Yeah, the above is in pseudo code, sorry for not making that clearer.
I write to a log, but the amount I'm writing is very limited - about 20 characters per name. Just enough to give me an idea as to how far through the processing I get before it dies. nohup.out receives this output.
My script is long and awkward, so I rather not post it in any detail.
I had a look at merlyn's column; I actually came across that code before and used to indicate the progess of a file upload. I also think the problem is some kind of timeout. I wonder could it be a stack overflow? I don't think so. Basically I'm using a hash and incrementing it with each occurrence of this name, so I'm not storing much on the stack. What's taking the script so long to process a name is the fact that it sometimes has to GET thousands of pages per name.
I'm thinking a fix might be to write a script which issues a ps -eaf. If the script is still running it does nothing, else it executes the script again. I could execute this second script ever hour using a cronjob.
Not very elegant, but certainly better than what I have right now.
| [reply] |