If GB's of data needs to be put in Hashes how can i use them effectively??

SireeshaYalamanchili has asked for the wisdom of the Perl Monks concerning the following question:

Hi, We have developed our project in Active Perl.In one Module we need to Stitch many files together as one XML file(with GB's of data). For this Stitching we are holding the data into Hashes and writing them on to XML file. But once the data is written into XML then also the program is not aborting ,it is taking lot much of time after finishing the process and then aborting.. and when we check the Task Manager the process running thier is the System Ideal Process.Is this problem is because of the Hashes or something else??. Please help us out in this and show us the solution. Thanks Sireesha

Comment on If GB's of data needs to be put in Hashes how can i use them effectively??

Replies are listed 'Best First'.
Re: If GB's of data needs to be put in Hashes how can i use them effectively?? by Joost (Canon) on Oct 24, 2006 at 11:24 UTC
It's probably the garbage collector working very hard to figure out which objects to free. If you're sure you're completely done, I think you can do `POSIX::_exit()` to force a quicker exit. See exit and _exit. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply] [d/l]
Re: If GB's of data needs to be put in Hashes how can i use them effectively?? by eyepopslikeamosquito (Archbishop) on Oct 24, 2006 at 11:30 UTC
when we check the Task Manager the process running thier is the System Ideal Process Do you mean the System Idle Process? It is normal for the System Idle Process to show a high value in the CPU column when nothing else is running. If you want to see it lower, try running SETI@home. ;-) Please help us out in this and show us the solution Please help us out in this and show us the code.	[reply]
Re: If GB's of data needs to be put in Hashes how can i use them effectively?? by Tanktalus (Canon) on Oct 24, 2006 at 16:05 UTC
"Doctor, it hurts when I raise my arm like this." "Then don't raise your arm like that." You have GB's of data ... and you want it in a single XML file? That's gonna suck no matter what you do with it. Perhaps you should use some sort of database, whether it's GDBM or some RDBMS (DB2, Oracle, MSSQL, MySQL, PostgreSQL, etc.). It'll be easier to build and easier to access... (If it's hierarchical data rather than relational, I think DB2 v9 has the ability to store the XML data as hierarchical data - "PureXML"(TM) - making it simpler to use without changing your algorithms very much. I imagine that most other DBMS's either have something similar, or will have something similar Real Soon Now(TM).)	[reply]
Re: If GB's of data needs to be put in Hashes how can i use them effectively?? by Anonymous Monk on Oct 24, 2006 at 11:07 UTC
It may be the hashes, it may not be. Hard to tell without code, and even with code, your problem description is vague. I don't think I understand what the behaviour of the program is.	[reply]