Hi, I have just registered to the site and this is my first post. I am Perl newbie. Recently wrote a script to create an extract out of the existing extract. The requirement is to pickup few columns as specified by client. It sounds simple but here is the twist... we have to lookup more than one extract to create a final file. For example: account file will contain 100 columns , get few from main account file it. Then for the corresponding account's client, get client details from another file. This way we have to lookup 4 to 5 different files for complete information. The design we suggested - Have a configuration file for fields information. In that specify for a field - file name, which column to be extracted from this, if this main file or lookup file, if lookup then lookup key.
ACCT NUMBER|positionfile.delim|2|6
ACCT SHORT NAME|accountfile.delim|6|6
PO TYPE|positionfile.delim|2|3
CUSIP|securityfile.delim|8|5
LOC CODE|positionfile.delim|2|47
LOC NAME|locationfile.delim|47|4
How to read it - at the start it is specified main file is account file. start reading that file line by line Then for ACCT NUMBER read main file, second column in main file is primary key for looking up positionfile.delim. Grep that and extract 6th column. Similarly LOC NAME, get the 47 column from main file, use that value as key in locationfile.delim and extract 4th column. and so on... The main problem is though I have written a working code for this, the speed is lethargic. Please suggest how to improve it or any better design. Delimter is pipe which is provided from another config file to support various types of delimited files. 


###Reading layout
        while (<$CONFIG>){
                chomp;
                my @list=split(/$hash_ref->{DELIMITER}/);
                $file_list = $list[1];
                $file_list =~ s/CCYYMMDD/$date/g;
                push @{$look_info{$list[0]}},$file_list."-".$list[2]."
+-".$list[3];
                push (@header, $list[0]);
        }

###reading data file

        while (<$SRC_FILE>){
                @data=split(/$hash_ref->{DELIMITER}/,$_);
                while ( ( my $column) = each %look_info){
                        foreach my $pos_info (@{$look_info{$column}}){
                                @look_here = split(/\-/, $pos_info);
                                #print "look here : @look_here \n";
                                $lkp_file_name = $look_here[0];
                                $key_location = $look_here[1];
                                $data_location = $look_here[2];
                                $key_string = $data[$key_location-1] ;
                                chomp $key_string;
                                print "key_string : $key_string \n";
                                my $key_pattern = "|".$key_string."|";
                                #print "key string is : $key_pattern \
+n";
                                $final_data = ` grep "$key_pattern" "$
+{indir}/${lkp_file_name}" |cut -d"|" -f"$data_location"`;
                                #print "data value is : $final_data \n
+";
                                chomp $final_data;
                                push @{$final_record{$column}},$final_
+data."|";

                        }
                }
        }


###printing final file

        my $i =0;
        my @currdata;
        $headerstring = join('|', @header);
        print $OUTPUT_FILE $headerstring."\n";
        while(1){
                foreach my $head (@header){
                        chomp $head;
                        $head =~ s/\\//g;
                        @currdata = @{$final_record{$head}};
                        print $OUTPUT_FILE "$currdata[$i]";
                }
                print $OUTPUT_FILE "\n";
                $i++;
                if ($i>($#currdata)){
                        last;
                }
        }

[download]

In reply to Perl Script performance issue by Tara

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.