in reply to out of memory exception

It is very interesting that the date format in your log suddenly changed: 

2010-03-30 10:18:35     Script started
2010-03-30 10:18:35     Date being processed: 2010_03_18>
2010-03-30 10:18:35     Starting to gather data
2010-03-30 10:18:35     Gathering nightly data
2010-03-30 12:32:19     Inserting data to database

2010-03-30_10-18-35:    Script started
2010-03-30_10-18-35:    Date being processed: 2010_03_18
2010-03-30_10-18-35:    Gathering nightly data . . .
2010-03-30_12-32-19:  Inserting data to database

[download]

Underscore instead of space; '-'s instead of ':'s; the disappearance of the '>' from the 'Date being processed' line. And this all starts right at the start of your script before you read your data. Somebody changed something in your logging. Look there first.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"I'd rather go naked than blow up my ass"

Replies are listed 'Best First'.
Re^2: out of memory exception
by siddheshsawant (Sexton) on Mar 30, 2010 at 21:41 UTC
    It was a typo !!!!
[reply]

      You should really copy&paste not retype such information. It would be interesting to see the real timestamps before and after the download. That might give an indication of how much more data was downloaded when it runs out of memory relative to when it doesn't.

      The obvious--though not necessarily feasible--solution would be to populate the database as you download the data, rather than accumulating it in a hash and then populating. LWP has a callback interface, see :content_cb that would allow you to get the data in chunks and store it on the fly, thereby avoiding the accumulation of data in memory.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom