I got the problem why the script was running very slow.

Actually, there is no problem with the XML validation part, problem was with the XML generation module which depends on a hash typically containing 1M entries in it.

Each and every record needs to be validated before pushing it to an external module, something like 
for record ( list of records )
do
create xml file with 1 record of data
validate the above xml file
if validated
then
  push record to success file
else
  push record to failure file
fi
done

use success file to generate final xml file

[download]
So, for each and every xml file generation with 1 record of data ( for validation ) will create a hash with 1M entries, that is the reason the script is performing so bad.

Now the question is - is there any way to 'pin' the created hash structure to memory so that any process making use of the data can refer to it using some 'memory namespace'.

In case of XSD parsing, such a file caching operation is possible using File::Cache, is there anything similar to that available?

If I could pin hash data structure to memory without re-creating it each and every time, there would be a mega improvement in the performance of my script.

Thanks in advance ! :)

In reply to Re: Improving script that uses xmllint to validate by matrixmadhan
in thread Improving script that uses xmllint to validate by matrixmadhan

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.