Hi monks,

I hope you all doing well. I have to write automation script for pulling data from greenplum, process the pulled data and generate report as an excel sheet in perl.
Currently I am embedding sql code on shell script and generating csv file. Once the csv file is genretaed we are manually format it and generate report as as excel sheet. These all task I am doing in unix server. For generating csv file of around 28 lacks (around 3 millions) it took 12 to 15 min.

Now few things which I want to discuss here is :-
1. How efficient would be the perl script for this task.
2. If I would wrote whole things like pulling data, processing and generating report would take more time or shall I break it half into shell and half in perl.
3. For generating report in excel I am planning to use spreadsheet::WriteExcel module or any other module are there in cpan which is more suitable then this module.
4. Is there any other way or technology through which I can proceed.

Thanks in advance for your kind advice and feedback

In reply to Automation for pulling huge data from greenplum database in perl. by sarf13

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.