c0bra has asked for the wisdom of the Perl Monks concerning the following question:

Hey,

I'm having trouble with substr not breaking up strings correctly.

Don't ask why, but I'm having to take a bunch of values, create a big delimited string and then break that string up into 100 character chunks and insert those chunks into a database. I know it's horrible and ugly but I don't have any other options.
There's two systems we're running this on. One is our development webserver and the other is our production server. Everything appears to be working fine on development but twice now on our production webserver we have had failures with one of the chunks being over 100 characters.

The strange thing is that we do a length() check before it inserts and it passes that too. The insert into the database failed of course and I did a character count on the failed chunk. It was 102 characters long. I tried using the same string on our development server and it worked. I also tried running the same code as a command-line perl script and that worked too.

Both webservers are running the same software, which is, albeit, old versions of Apache/Stronghold, mod_perl and perl (5.005_03).

Here's the code that I'm using. @valarray contains the list of fields. @elems will contain the chunks. 

    my $valstring = join('|', @valarray);
    
    my @elems = ();
    my $len = 100;
    my $count = 0;
    do {
        my $string = substr($valstring, 100*$count, 100);
        push(@elems, $string);
             
        $count++;
        $len = 100*$count;
    }
    while ($len < length $valstring);

[download]

I don't know if this being caused by strange characters. Could that be it?

Replies are listed 'Best First'.
•Re: Strange substr Problem
by merlyn (Sage) on Mar 13, 2003 at 15:44 UTC
[reply]
[d/l]
      Well that's certainly a lot less code, but if there is a problem existing between substr and the data that is going in won't I still get errors?
[reply]
Re: Strange substr Problem
by c0bra (Acolyte) on Mar 13, 2003 at 18:18 UTC
    I found the problem. It's because we had to expand ampersands into the word "and" because of our worthless payment engine, and those transactions had a field with 1 & in them.

    *sigh*
[reply]

      Glad you found your problem, but a note of caution. If your ever going to upgrade from 5.005 to a newer version of Perl, and if there is any chance that any of your data will contain utf-8 characters, substr might cause you problems in that it deals in characters not bytes.

      You might consider using unpack 'a100' or use bytes to avoid problems as and when you upgrade. A snippet to demonstrate the potential problem and both solutions. 

      #! perl -slw
use strict;
use 5.008;

my $utfdata = join'',map{ chr(0xffff + $_) } 1 .. 100;
print length $utfdata;                      #! Gives 100

my @bytes = unpack 'C*', $utfdata;
print scalar @bytes;                     #! Gives 400

my $chunk = substr $utfdata, 0, 100;
print length $chunk;                     #! Gives 100

@bytes = unpack 'C*', $chunk;
print scalar @bytes;                     #! Gives 400

{
    use bytes;
    print length $utfdata;               #! Gives 400
    print length $chunk;                 #! Gives 400
}

my @chunks = unpack '(a100)*', $utfdata;
print scalar @chunks;                    #! Gives 4
print "@{[ map{length} @chunks ]}";      #! Gives 100 100 100 100

      [download]
      Examine what is said, not who speaks.
      1) When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
      2) The only way of discovering the limits of the possible is to venture a little way past them into the impossible
      3) Any sufficiently advanced technology is indistinguishable from magic.
      Arthur C. Clarke.
[reply]
[d/l]
Re: Strange substr Problem
by pg (Canon) on Mar 13, 2003 at 16:02 UTC
    Your code looks just fine, doesn't matter whether it looks perlish or not.

    I suggest you to study the actually content of your string, by using something like ord().

    I am not saying garbage characters will make your code go wrong, but it would confuse you, as an observer, in various ways.
[reply]
Re: Strange substr Problem
by zby (Vicar) on Mar 13, 2003 at 16:28 UTC
    When the database uses a different encoding than the interpreter then the data passed to it is recoded by the library. This behaviour is dependant on the used library and database engine - so you need to be more specyfic.
[reply]
      We are running Oracle 8.1.7.4.0 on both prod and dev.
[reply]

Back to Seekers of Perl Wisdom