Thanks
Where does the data come from?
Is it all loaded up from a file at start-up? Or is it built-up over time as the process(es) run? Other?
====>From database, do some transform. Once done, never change.
Could you post a few (2 or 3) examples of the keys + associated values?
====>Dont have it for now. value is mixed combination of hash/array/refs of hash,of array, of refs.
Is the entire dataset required for every iteration?
====>yes.
What do you do with the data when the program(s) end?
====>discards. after the program end , i will get a new data, and insert/update to database.
Do you save it some where? Discard it and re-build it next time the program runs?
====>mysql
Or is the data-structure built once as you process each log file, some statistics derived from it and then discarded never to be used again?
====>see above.
Thanks.~
| [reply] |
Where does the data come from?
Is it all loaded up from a file at start-up? Or is it built-up over time as the process(es) run? Other?
====>From database, do some transform. Once done, never change.
Given the scant description of the data and processing, my only response is that rather than filling (more than) your memory capacity with data read from an RDBMS in order to do processing in Perl, you should be leaving the data in the DB and formulating SQL queries that allow the RDMBS to perform the processing for you.
Maybe if you explained why that is not possible, then an alternative solution might suggest itself, but it would require much more detail.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |