How to find out, why my perl code is slow.

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I have written a (working) perl program, but it is quite slow. I have troubles to find out why. Is there a better way to find out why, than guess and try ?? How do you start a speed analyses to find out why a program is slow ?

Thanks for your help !!!

My program reads a complete binary file into $f (Not show in the code fragment). Than it decodes following structure:

The first byte is the first data byte. (range from 0 to 59, 255 means datastream is over)

The second byte defines how many "position bytes" bytes are following. (binary: 1=position byte follows, so 1 to max 8 position bytes, 0=position byte equal to 0x00 skipped)

The position bytes define which data bytes are changed in this data set. (binary: 1=data byte changed, so max 64 data bytes)

Now the databytes follows (one for each binary 1 in a position byte), followed by a CRC byte of this dataset.

Than the next data set is comming, until 255 is following.

The program checks the data and does statistics. (Not shown in my code fragment here. Also when this part is commented out, the program is too slow !,)

Hope this description was not too short for this data structure.

Please find my program here:

$crc=255; 
$data[0]=readByte();
do
  { $_=sprintf("%08b",readByte());                                    
+            # BYTE1
    $byte2='';  
    while (m/(.)/g)
    { if ($1 == 0) { $byte2.='00000000';}
      else         { $byte2.=sprintf("%08b",readByte());}             
+            # positions bytes
    } 
    $data[pos($byte2)]=readByte() while ($byte2=~m/(1)/g);            
+            # put data value into @data on the correct position 
    readByte(); if ($crc>0) {print "ERROR:  CRC (POS:".pos($f).')'; ex
+it; }       # crc check
    $crc=255;
    # ---- USE DATA => DELETED --------
    
    $data[0]=readByte();
  } while ($data[0]<255);
}

sub readByte
{ unless ($f=~m/(.)/gs) {print "Last data structure not complete !"; e
+xit;}
  my $x=ord($1);
  $crc^=$x;                                                 # CRC
  for my $r(1..8)
  { if (($crc&1)>0) {$crc=($crc>>1)^0x8c;}
    else            {$crc=$crc>>1;}  
  }
  return ($x); 
}
[download]

Comment on How to find out, why my perl code is slow. Download Code

Replies are listed 'Best First'.
Re: How to find out, why my perl code is slow. by Laurent_R (Canon) on Nov 04, 2018 at 23:05 UTC
Can you please tell us what you mean exactly by slow? Or, in other words, how many bytes (or megabytes, whatever) you're processing per unit of time? And, if possible, how much faster your would expect tor want) it to be? For finding the slow parts in your program, the best is probably to use a profiling tool such as Devel::NYTProf. Otherwise, as a general comment, I would suspect that using the regex engine in a loop to fetch one byte at a time is probably quite inefficient. unpack is very likely to be significantly faster if you can use it. Even substr in a loop is quite probably going to run faster.	[reply]
Re^2: How to find out, why my perl code is slow. by Anonymous Monk on Nov 05, 2018 at 06:04 UTC
An file has a size of about 420 kbytes, so about 55000 to 65000 data sets. Processing (Decoding) one Files takes in average 1.75 seconds (without data post processing). Usually at one step between 50 (~1:27) and 400 files (~11:37) are processed. It would be nice to be with 400 files clear below one minute.<\p> I will have a look to MDevel::NYTProf ......<\p>	[reply]
Re: How to find out, why my perl code is slow. by choroba (Cardinal) on Nov 04, 2018 at 21:07 UTC
I'm not sure whether I understand your description correctly. Could you include a (hexdump of) short data sample? Using unpack or pack should be faster and more suitable for the task, but I don't have enough will to decipher your code and blindly post my guess. ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l]
Re^2: How to find out, why my perl code is slow. by Anonymous Monk on Nov 04, 2018 at 21:26 UTC
Please find 3 example datasets: (set 1 starts with 0x31, set 2 starts with 0x32, ser3 with 0x33) `$f=(0x31, 0xFC, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFE, 0x58, 0x17, 0x30, +0x06, 0x18, 0x26, 0x25, 0x27, 0x26, 0x21, 0x23, 0x21, 0x6E, 0x5D, 0x5 +3, 0x46, 0x17, 0x2D, 0x62, 0x00, 0x00, 0xC0, 0x00, 0x00, 0x00, 0x00, +0x00, 0x00, 0x00, 0x00, 0x26, 0x2F, 0x2D, 0x2D, 0x2D, 0x2E, 0x2D, 0x0 +0, 0x00, 0x80, 0x00, 0x00, 0x01, 0x50, 0x59, 0x12, 0x00, 0xA8, 0x32, +0xC0, 0x01, 0xD0, 0x26, 0x25, 0x22, 0x22, 0x89, 0x33, 0x60, 0x80, 0x8 +0, 0x26, 0x18, 0x7C, 0x34, 0xFF);` [download]	[reply] [d/l]
Re^3: How to find out, why my perl code is slow. by Anonymous Monk on Nov 04, 2018 at 21:34 UTC
Sorry 1 was too fast. The 0x34 at the end should not be there ! (start of new dataset !) Correct is: `$f=(0x31, 0xFC, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFE, 0x58, 0x17, 0x30, +0x06, 0x18, 0x26, 0x25, 0x27, 0x26, 0x21, 0x23, 0x21, 0x6E, 0x5D, 0x5 +3, 0x46, 0x17, 0x2D, 0x62, 0x00, 0x00, 0xC0, 0x00, 0x00, 0x00, 0x00, +0x00, 0x00, 0x00, 0x00, 0x26, 0x2F, 0x2D, 0x2D, 0x2D, 0x2E, 0x2D, 0x0 +0, 0x00, 0x80, 0x00, 0x00, 0x01, 0x50, 0x59, 0x12, 0x00, 0xA8, 0x32, +0xC0, 0x01, 0xD0, 0x26, 0x25, 0x22, 0x22, 0x89, 0x33, 0x60, 0x80, 0x8 +0, 0x26, 0x18, 0x7C, 0xFF);` [download]	[reply] [d/l]
Re^4: How to find out, why my perl code is slow. by jwkrahn (Abbot) on Nov 05, 2018 at 06:53 UTC
Re: How to find out, why my perl code is slow. by harangzsolt33 (Deacon) on Nov 05, 2018 at 02:32 UTC
Looking at your code, I see several problems that may be slowing it down. First of all using the . to merge two strings does slow down things a lot! Secondly, you use sprintf to convert a byte to binary format, which is unnecessary, I think. You could go around that and just perform the necessary operations without converting the number to binary. Please try to do that! Your code will be a lot faster! Also, you are putting a regex in while loop's header. Anytime you use regex, it could slow down the code, especially if you put it into a loop that is supposed to go through your code byte-by-byte. Anytime you build a program that runs through some long code byte by byte, then you really have to make it fast. Don't use regex, don't use the "." or ".=" operators. Don't convert back and forth between bases unnecessarily. I see that you are also using the ord() function. That is slower than using vec(). vec() takes three arguments. The first one is the string or character you want to get the byte value from. The second arg is the pointer, and the third arg is usually 8, which tells it to grab an entire byte. Example: $a = vec("Thank you", 0, 8) will grab the letter "T" whose ASCII value is 84, and so $a will contain 84. You can then test if any bits in 84 are on or off using the & operator. if ($a & 1) then you are checking the lowest bit. if ($a & 2), you are checking if the second bit is on. if ($a & 4), you are checking the third bit, and so on... then you can do some arithmetic or bitwise operation on those bits if you want to and plug the new value into a string. I have found that manipulating strings using the vec() function is a lot faster than working with arrays. Creating an array of 10 elements will also take up more memory than creating a string that is 10 bytes long. You can address any byte or word in a string as if it were an array using the vec() function. vec("ABCDEF", 0, 16) will pretend that this string is an array of words, and it will grab the first word. (A word is a 16-bit integer.) So, it will grab "AB" and the value of AB is 0x4142, which is 16706. vec() can also be used to overwrite a string without modifying its length. Every time you use the "." operator to merge strings, it's going to copy strings in the memory and possibly reserve more memory and then free up some memory. When you use vec() to write to a string, it is very fast! Here is an example : $a = 'hello!'; vec($a, 0, 8) = 72; This code simply overwrites the first byte of string $a. It replaces the first byte with a capital "H." It's kind of like using pointers in C language. This is a very cool feature in Perl. So, whenever you want to take a value from a string and modify it and write it back, use vec(). Use vec() whenever possible! It makes your code faster!	[reply]
Re^2: How to find out, why my perl code is slow. by Anonymous Monk on Nov 05, 2018 at 18:31 UTC
Thanks for your recommendations !! I modified a lot in my code, it is now about 6 times faster !: * use vec to get the byte from the file string. * removed all "." or ".=" operators in often used places * coded position bytes new (use "&" instead of regex,binary strings and no loops) * pre calculated "for loop of crc" and stored it into a look up table (256 Bytes). I modified step by step. But: My version with arrays instead of using strings and vec is faster (e.g crc-look up table, no change of array size over the program). Could it be that this is different in win, which I use, and linux ? In the Profile of -MDevel::NYTProf I can see a big time at the last statement of a subroutine. e.g. 3.8 seconds at the last statement ($fpos++;) of readByte() @ 1.700.000 calls. This time is nearly independent of the statement itself. Inserting a return makes it slower: 4s for the return. The statement is than rated as fast(200ms). Where does this time come from ? Following is not clear for me: I tried to insert the content of the subroutine into the code instead using this subroutine. This version is in the profiler 15% faster, but comparing this two versions without profiler, it shows less than 1% speed difference. So not worth doing it. Any idea what happens here ?	[reply]
Re^3: How to find out, why my perl code is slow. by haukex (Archbishop) on Nov 05, 2018 at 18:48 UTC
the last statement ($fpos++;) of readByte() ... Any idea what happens here ? You haven't shown us the code, so we don't know... could you provide a Short, Self-Contained, Correct Example that we could run ourselves? (e.g. with some sample input data)	[reply]
Re^4: How to find out, why my perl code is slow. by Anonymous Monk on Nov 05, 2018 at 19:42 UTC
Re^5: How to find out, why my perl code is slow. by Corion (Patriarch) on Nov 05, 2018 at 19:53 UTC
Re^3: How to find out, why my perl code is slow. by swl (Prior) on Nov 06, 2018 at 00:54 UTC
I've found similar differences in the past between profiling and benchmarking. I assume it is because profiling is run under the debugger and so some optimisations are disabled or there is more bookkeeping involved. Running both profiler and benchmarking is always a sensible process. As for the last statement taking time, I think this is because it includes the time for the various bookkeeping operations run at the end of the subroutine (e.g. memory cleanup). If you end the sub with "`1;`" then you should see the actual time taken for the increment (although it might break your program). Hopefully others will be better able to comment on or clarify the above.	[reply] [d/l]
Re: How to find out, why my perl code is slow. by Anonymous Monk on Nov 04, 2018 at 21:52 UTC
How do you start a speed analyses to find out why a program is slow ? use Devel::NYTProf	[reply]
Re^2: How to find out, why my perl code is slow. by swl (Prior) on Nov 05, 2018 at 01:36 UTC
Definitely use Devel::NYTProf. The perl documentation also contains a section about this topic that should be read. See https://perldoc.perl.org/perlperf.html. The documentation for Devel::NYTProf gives more details about how to use it specifically. An internet search will produce plenty of examples, but one that includes example output is at https://www.effectiveperlprogramming.com/2011/10/profile-with-develnytprof.	[reply]
Re: How to find out, why my perl code is slow. by Marshall (Canon) on Nov 05, 2018 at 19:15 UTC
I was confused by your textual description of the binary format as well as your code. In general when processing binary files, avoid regex. Your friends should be: substr,ord,unpack and pack. Reading a 500K binary file into a single var using binmode should be fine. Below is a simple example, use unpack for multi-byte substrings. You can account for the difference between Intel byte order and Motorola byte order. The relatively low level functions that I mentioned will "run like a rocket". #!/usr/bin/perl use strict; use warnings; my $f = "31FCFFFFFFFFFFFE5817300618262527262123216E5D5346172D620000C00 +000000000000000262F2D2D2D2E2D00008000000150591200A832C001D02625222289 +3360808026187CFF"; $f = pack("H*",$f); # substr EXPR,OFFSET,LENGTH,REPLACEMENT print "Number of bytes: ", length $f, "\n"; printf "record Len? : %x hex \n",ord(substr($f,0,1)); print "record Len? : ", ord(substr($f,0,1))," decimal \n"; print "not right first byte, ASCII '1':", substr($f,0,1),"\n"; ##ASCI +I "1"; __END__ Prints: Number of bytes: 73 record Len? : 31 hex record Len? : 49 decimal not right first byte, ASCII '1':1 [download] I would be thinking of a subroutine that you call like: `my $next_byte_index = process_record (\$data, $start_byte_index);` [download] passing a reference to the data instead of the data itself will speed things up quite a bit. To incorporate this into a loop, you will have to consider the "end conditions", when the loop ends and the difference between the length in bytes vs the "off by one" index number.	[reply] [d/l] [select]