in reply to Re^4: using system command in regex
in thread using system command in regex

Of course it is dead slow. You are opening, reading and closing each file for 10 * 60 = 600 times to get the sum for each second. You should read each file once and store the sums in a hash, keyed by the timestamp, like so:

my %SMPP_count; foreach my $file (@files) { open (FILE,"$file"); while(<FILE>) { next unless /\b(?:GSM|SMPP)\b/; # avoid uninteresting +lines chomp; my @ary = (split /\s|\|/, $_) [3,21,24]; my $time = (split ' ', $ary[0])[4]; # first element is +t timestamp, right? $SMPP_count{$time}++ if $ary[0] eq $stamp and $ary[1] + eq "Submit" and $ary[2] =~ /(GSM|SMPP)/; }; } # now iterate over the keys of the hash to make up your sums for my $time ( sort keys %SMPP_count) { my $sum = $SMPP_count{$time}; ... }
perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'

Replies are listed 'Best First'.
Re^6: using system command in regex
by ravi45722 (Pilgrim) on Oct 15, 2015 at 10:41 UTC

    What an Idea pragrammatic.... Nearly from three days struggling with this. Finally I solved it. Working great. Thank you very much. And I want to ask you one more thing. As already told I have three (Postpaid, Prepaid, Delivery) files for every 10 minutes. After calculating I am uploading all the values into DB like this.

    Date Hour Mo_resp MT_resp AO_resp Percentage 10-08-2015 00:00 256 382 36 87% 10-08-2015 00:10 491 438 12 92%

    (Its a sample. Actually my DB contains 38 columns) Like this I am uploading all values for every 10 minutes. Now my requirement is to add all MO_resp all AO_resp ... and son on (all columns) which occurred in 00 hour to write an hourly report in excel sheet. For that I am Doing

    use DBI; my $hour_db = DBI->connect("DBI:mysql:database=$db;host=$host;mysql_so +cket=/opt/lampstack-5.5.27-0/mysql/tmp/mysql.sock","root","", {'Raise +Error' => 1}); my @column_names = ("MO_resp","MT_resp","AO_resp"); foreach my $column_name (@column_names) { my $hour_sth = $hour_db->prepare("Select sum($column_name) from $t +able_name where Date='$db_date' and Hour like='$hour:%'"); $hour_sth->execute() or die $DBI::errstr; ..... }

    #Like this I am reading each column sum one by one. But I feel this is not a good method. Can you show me a way???

      First, I have to second marto in that you should always use placeholders, third (since second is already used) you could aggregate all your columns into one call:

      my @column_names = ("MO_resp","MT_resp","AO_resp"); my $sql = "select ".join ",", map { "sum($_)" } @column_names; $sql .= " from ? where Date = ? and Hour like ?"; my $hour_sth = $hour_db->prepare( $sql ); $hour_sth->execute($table_name, $db_date, "$hour:%") or die $DBI::errs +tr;

      See join and map.
      If you are iterating over $db_date and $hour, you should keep the call to $hour_db->prepare outside those loops to reduce the overhead of binding placeholders in the SQL statement.

      perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'

        Hey Programatic you are also doing the same but you are passing all the column names once by using join & map. Good thought. But in place holders If i place table name its showing a error.

        DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''CDR' where Date = '13-10-2015' and Hour like '00:%'' at line 1 at DB_Test.pl line 25.

        So I replaced my '?' with table name. Then its working good. Even though I used your logic I need to maintain all my column names in array. I want to change that. Here I am writing all columns in program into big arrays which is looking odd. Can I get that coloumn names from db???

        Sample (Tested)

        #!/usr/bin/perl use strict; use warnings; use DBI; my $db_date=$ARGV[0]; #04-10-2015 my $hour = "00"; my $db_name = "ravi"; my $table_name = "CDR"; my $host = "xxx.xx.x.xxx"; my @CDR_array = ("Total_MO","Total_MO_sucess","MO_userdep_error","MO_s +ubcrib_error","MO_system_error","Total_MO_success_rate","Total_AO","T +otal_AO_sucess","AO_userdep_error","AO_subcrib_error","AO_system_erro +r","Total_AO_success_rate","Total_MT_P2P","Total_MT_sucess_P2P","MT_P +2P_userdep_error","MT_P2P_subcrib_error","MT_P2P_system_error","Total +_MT_P2P_success_rate","Total_MT_A2P","Total_MT_sucess_A2P","MT_A2P_us +erdep_error","MT_A2P_subcrib_error","MT_A2P_system_error","Total_MT_A +2P_success_rate","Total_AT","Total_AT_sucess","AT_ESMEdep_error","AT_ +network_error","AT_system_error","Total_AT_success_rate","P2P_FDA_cou +nt","P2P_FDA_success_count","P2P_FDA_success_rate","A2P_FDA_count","A +2P_FDA_success_count","A2P_FDA_success_rate","Total_Count","Total_TPS +","Greatest_TPS"); my $dbh = DBI->connect("DBI:mysql:database=$db_name;host=$host;mysql_s +ocket=/opt/lampstack-5.5.27-0/mysql/tmp/mysql.sock","root","",{'Raise +Error' => 1}); my $sql = "select ".join ",", map { "sum($_)" } @CDR_array; $sql .= " from CDR where Date = ? and Hour like ?"; my $sth = $dbh->prepare( $sql ) or die "Cant get prepare"; $sth->execute($db_date, "$hour:%") or die $DBI::errstr; my @my_array = $sth->fetchrow_array(); print $my_array[1],$/; print @my_array,$/;
      plus you seem to have misunderstood the first sentence of this. "programmatic" is a quality of his nick, see e.g. there :-)

        My nick feels honoured being linked to something as old as FORTRAN - but the concept is much older; it was invented after speech and before writing, and eventually led to an early cosmic calculator (though at that time writing already had been invented, I guess).

        I have heard (through memory sharing) that australian aborigines still have a vast block of shared memory, which they access entirely wireless.

        Before the concept of shmem, there was fork.

        perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'