in reply to Remote database and process

What would you being doing with the millions of records once you received them?

Eg. Processing them immediately or storing them to local disk for further processing?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy

Replies are listed 'Best First'.
Re^2: Remote database and process
by ashokpj (Hermit) on Jun 07, 2010 at 07:38 UTC
    Hi, Am planning to localfile for further processing.
[reply]

      Then I would suggest you look at the mysql --quick command & option.

      If you combine that with the mysql < script.sql > results., the memory usage by the local process will be minimal and the results will be retrieved and filed far more quickly than you will ever achieve with Perl & DBI. (Sorry if that offends anyone, but it is true!)

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
        (Sorry if that offends anyone, but it is true!)

        Disclaimers are so offensive

        :p

[reply]
        Hi, I wonder if POE::Component::DBIAgent - POE Component for running asynchronous DBI calls is efficient to do? Please sugguest.
[reply]
Re^2: Remote database and process
by ashokpj (Hermit) on Jun 07, 2010 at 07:38 UTC
    Hi, Am planning to write to local file for further processing.
[reply]

In Section Seekers of Perl Wisdom