I have this Perl script which first fetches one field and its corresponding value from a 1gb(around 150000 lines) input file and stores into one hash. now after processing this file it takes another 16gb input file (around 30 million lines), reads line by line, compares it with the hash keys and print the corresponding hash value to the output file with few other required data, creating a 2gb output file. Now they have asked me to add 4 new fields into the report which needs 4 more hashes to be included into the script. but when I added and tested the script...it caused severe memory issue in the server.. current memory details on my linux server.

 # free
             total       used       free     shared    buffers     cac
+hed
Mem:       8158176    8124068      34108          0      13248    7928
+420
-/+ buffers/cache:     182400    7975776
Swap:      2104472     103976    2000496

[download]

I need to know how can I resolve this memory issue...If needed, how much more ram I need to add into my server...I know information provided by me could be vague for you.. but please let me know what else you need to know (shall I paste the complete script here?)...please help...

In reply to How to reslove this memory issue by Ankur_kuls

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.