in reply to How to find out, why my perl code is slow.

Looking at your code, I see several problems that may be slowing it down. First of all using the . to merge two strings does slow down things a lot! Secondly, you use sprintf to convert a byte to binary format, which is unnecessary, I think. You could go around that and just perform the necessary operations without converting the number to binary. Please try to do that! Your code will be a lot faster! Also, you are putting a regex in while loop's header. Anytime you use regex, it could slow down the code, especially if you put it into a loop that is supposed to go through your code byte-by-byte. Anytime you build a program that runs through some long code byte by byte, then you really have to make it fast. Don't use regex, don't use the "." or ".=" operators. Don't convert back and forth between bases unnecessarily. I see that you are also using the ord() function. That is slower than using vec().

vec() takes three arguments. The first one is the string or character you want to get the byte value from. The second arg is the pointer, and the third arg is usually 8, which tells it to grab an entire byte. Example: $a = vec("Thank you", 0, 8) will grab the letter "T" whose ASCII value is 84, and so $a will contain 84. You can then test if any bits in 84 are on or off using the & operator. if ($a & 1) then you are checking the lowest bit. if ($a & 2), you are checking if the second bit is on. if ($a & 4), you are checking the third bit, and so on... then you can do some arithmetic or bitwise operation on those bits if you want to and plug the new value into a string. I have found that manipulating strings using the vec() function is a lot faster than working with arrays. Creating an array of 10 elements will also take up more memory than creating a string that is 10 bytes long. You can address any byte or word in a string as if it were an array using the vec() function. vec("ABCDEF", 0, 16) will pretend that this string is an array of words, and it will grab the first word. (A word is a 16-bit integer.) So, it will grab "AB" and the value of AB is 0x4142, which is 16706.

vec() can also be used to overwrite a string without modifying its length. Every time you use the "." operator to merge strings, it's going to copy strings in the memory and possibly reserve more memory and then free up some memory. When you use vec() to write to a string, it is very fast! Here is an example : $a = 'hello!'; vec($a, 0, 8) = 72; This code simply overwrites the first byte of string $a. It replaces the first byte with a capital "H." It's kind of like using pointers in C language. This is a very cool feature in Perl. So, whenever you want to take a value from a string and modify it and write it back, use vec(). Use vec() whenever possible! It makes your code faster!

  • Comment on Re: How to find out, why my perl code is slow.

Replies are listed 'Best First'.
Re^2: How to find out, why my perl code is slow.
by Anonymous Monk on Nov 05, 2018 at 18:31 UTC

    Thanks for your recommendations !!

    I modified a lot in my code, it is now about 6 times faster !: * use vec to get the byte from the file string. * removed all "." or ".=" operators in often used places * coded position bytes new (use "&" instead of regex,binary strings and no loops) * pre calculated "for loop of crc" and stored it into a look up table (256 Bytes).

    I modified step by step. But: My version with arrays instead of using strings and vec is faster (e.g crc-look up table, no change of array size over the program). Could it be that this is different in win, which I use, and linux ?

    In the Profile of -MDevel::NYTProf I can see a big time at the last statement of a subroutine. e.g. 3.8 seconds at the last statement ($fpos++;) of readByte() @ 1.700.000 calls. This time is nearly independent of the statement itself. Inserting a return makes it slower: 4s for the return. The statement is than rated as fast(200ms). Where does this time come from ? Following is not clear for me: I tried to insert the content of the subroutine into the code instead using this subroutine. This version is in the profiler 15% faster, but comparing this two versions without profiler, it shows less than 1% speed difference. So not worth doing it. Any idea what happens here ?

[reply]
      the last statement ($fpos++;) of readByte() ... Any idea what happens here ?

      You haven't shown us the code, so we don't know... could you provide a Short, Self-Contained, Correct Example that we could run ourselves? (e.g. with some sample input data)

[reply]

        Please find the (simplified) program below. The file is simulated with an array. It is running itself. 

        #!/usr/bin/perl 
use strict;
use warnings;
$\="\n"; $,=", ";

#------------------------------------------------
my($f, $fpos, $fposlen, @data, $byte1, @byte2, $crc, @crcval); 
initCRC();
ReadDatFile();
exit;

#------------------------------------------------
sub ReadDatFile
{ # simulate file read:
  # three datasets:  0,1,2,3,4,5,6,7,8,9  and  10,11,2,3,4,5,6,7,8,9 a
+nd 20,11,2,3,4,5,6,7,18,19
  my @file=(0x00, 0xc0,0xff,0x80,1,2,3,4,5,6,7,8,9,0x8e,   10,0x80,0x8
+0,11,0x9d,   20,0xc0,0x01,0x80,18,19,0x1c,   0xff);
  $f.=chr($_)|'' foreach (@file);
  
  $crc=255; $fposlen=length($f); $fpos=0;
  @byte2=(0,0,0,0,0,0,0,0);
  readByte(); $data[0]=$_;                              # first data
  
  do
  { readByte(); $byte1=$_;                              # BYTE1: $_
    if ($byte1 & 0x80) {readByte(); $byte2[0]=$_; } 
    if ($byte1 & 0x40) {readByte(); $byte2[1]=$_; }
    if ($byte1 & 0x20) {readByte(); $byte2[2]=$_; }
    if ($byte1 & 0x10) {readByte(); $byte2[3]=$_; } 
    if ($byte1 & 0x08) {readByte(); $byte2[4]=$_; }
    if ($byte1 & 0x04) {readByte(); $byte2[5]=$_; }
    if ($byte1 & 0x02) {readByte(); $byte2[6]=$_; }
    if ($byte1 & 0x01) {readByte(); $byte2[7]=$_; }

    do_byte2( 0) if ($byte1 & 0x80); 
    do_byte2( 8) if ($byte1 & 0x40); 
    do_byte2(16) if ($byte1 & 0x20);
    do_byte2(24) if ($byte1 & 0x10); 
    do_byte2(32) if ($byte1 & 0x08); 
    do_byte2(40) if ($byte1 & 0x04); 
    do_byte2(48) if ($byte1 & 0x02); 
    do_byte2(56) if ($byte1 & 0x01); 
    
    readByte(); if ($crc>0) {print "ERROR: CRC (POS:$fpos)"; exit; }  
+ # crc check
    $crc=255;     

    print @data;           # ---- USE DATA ---------
    
    readByte(); $data[0]=$_;                         # next dataset
  } while ($_<255);
  
}

sub do_byte2
{ my $bpos=$_[0];                                     # pos in data fr
+om byte0
  my $pp=$byte2[$bpos>>3];
  if ($pp & 0x80) {readByte(); $data[$bpos+1]=$_;}    # take value int
+o data array
  if ($pp & 0x40) {readByte(); $data[$bpos+2]=$_;}
  if ($pp & 0x20) {readByte(); $data[$bpos+3]=$_;}
  if ($pp & 0x10) {readByte(); $data[$bpos+4]=$_;}
  if ($pp & 0x08) {readByte(); $data[$bpos+5]=$_;}
  if ($pp & 0x04) {readByte(); $data[$bpos+6]=$_;}
  if ($pp & 0x02) {readByte(); $data[$bpos+7]=$_;}
  if ($pp & 0x01) {readByte(); $data[$bpos+8]=$_;}
}

sub readByte
{ if ($fposlen==$fpos) {print "Last data structure not complete !"; ex
+it;}
  $_=vec($f,$fpos,8); 
  $crc=$crcval[$crc^$_];         # CRC     
  $fpos++;  
}

sub initCRC
{ for my $v (0..255)
  { $_=$v;
    for my $r (1..8)
    { if (($_&1)>0) {$_=($_>>1)^0x8c;}
      else          {$_=$_>>1;}  
    }
    $crcval[$v]=$_;  
} }

        [download]
[reply]
[d/l]

      I've found similar differences in the past between profiling and benchmarking. I assume it is because profiling is run under the debugger and so some optimisations are disabled or there is more bookkeeping involved. Running both profiler and benchmarking is always a sensible process.

      As for the last statement taking time, I think this is because it includes the time for the various bookkeeping operations run at the end of the subroutine (e.g. memory cleanup). If you end the sub with "1;" then you should see the actual time taken for the increment (although it might break your program).

      Hopefully others will be better able to comment on or clarify the above.

[reply]
[d/l]

In Section Seekers of Perl Wisdom