Re: processing slows way down

One possibility is that you a have a badly scoped variable or structure that is retaining redundant information from record to record and growing to the point that it becomes harder and harder to allocated enough memory.

A fairly simple test would be to add a

my $recno = 0;
while( <GET RECORD> ) {
    ...
    AddRecordToDB( ... );
    do{
        print 'Check memory:'; 
        <STDIN>;
    } unless ++$recno % 1000;
}
[download]

at the bottom of the main loop and use ps or similar tool to monitor and record the tasks memory usage.

If the memory growth increases substaintially from one check to the next, your leaking somewhere. Then it's a case of tracking down where.

Examine what is said, not who speaks.

"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon

Comment on Re: processing slows way down Select or Download Code

Replies are listed 'Best First'.
Re^2: processing slows way down by Anonymous Monk on Jul 10, 2004 at 20:14 UTC
The fields for the user data are defined by my's at the top of the script. The data is read, falls into a while until end of file. In the while, the fields are interpreted, the table fields are formatted, the insert is done, repeat the while. We are not perl whizzes. The extent of our perl knowledge is pretty much what I explained above. We pick up bits of new bits of code as we come across new situations. The scope for what I described is through whole script because of the my's at the top of the script? I so, how could it be a scope issue? Thank you for trying to help.	[reply]
Re^3: processing slows way down by BrowserUk (Patriarch) on Jul 10, 2004 at 20:52 UTC
how could it be a scope issue? Without seeing the code, there really is no way to tell where the problem might lie. That's why I offered the suggestion on how you could check the problem out. It could be that there is a leak in one of the components your using to access Oracle. The simple test there is to comment out the DB code within the loop and re-run the script. If that stops the memory growth, you know where to look. If it doesn't, you also know where. All of which is pure speculation until you perform the test to see if memory leakage is the cause of the slowdown. Another possibility is that you are misusing DBI. There have been examples of code here at PM where people have been preparing their statements each time around the loop, rather than once outside and then executing them inside. Again, without sight of the code, pure speculation. Examine what is said, not who speaks. "Efficiency is intelligent laziness." -David Dunham "Think for yourself!" - Abigail "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon	[reply] [d/l]
Re^4: processing slows way down by Anonymous Monk on Jul 11, 2004 at 00:42 UTC
Ok, thank you BrowserUk. I will look into what you describe and try to get back here with what happened.	[reply]