in reply to Re: Reading (the same) data in different ways & memory usage
in thread Reading (the same) data in different ways & memory usage

I implemented your idea with a slight variation (moved the decimal-correction in the numeric data branch and put the check for empty fields in the non-numeric branch: 
        if ($self->datatype->[$_] =~ /^(?:TINYINT|MEDIUMINT|SMALLINT|I
+NT|INTEGER|BIGINT|FLOAT|DOUBLE)$/) {
            $hr_returnvalue->{$CurrentColumnName} = 0 + substr ($texti
+nput, $self->flatfield_start->[$_], $self->flatfield_length->[$_]);# 
+create a numeric value.
            # Decimal-correction
            if ($self->decimals->[$_] > 0 and defined $hr_returnvalue-
+>{$CurrentColumnName}) {
                $hr_returnvalue->{$CurrentColumnName} /= 10**$self->de
+cimals->[$_];
            }
        } else {
            $hr_returnvalue->{$CurrentColumnName} = substr ($textinput
+, $self->flatfield_start->[$_], $self->flatfield_length->[$_]);
            $hr_returnvalue->{$CurrentColumnName} =~ s/^\s*(.*?)\s*$/$
+1/;    # Trim whitespace
            # Fill empty fields with that field's default value, if su
+ch a value is defined
            if ($hr_returnvalue->{$CurrentColumnName} eq "") { 
                if (defined $self->standadefaultard->[$_]) {
                    if ($self->datatype->[$_] =~ /^(?:CHAR|VARCHAR|DAT
+E|TIME|DATETIME)$/) {
                        $hr_returnvalue->{$CurrentColumnName} = sprint
+f ("%s", $self->default->[$_]);
                    } else {
                        $hr_returnvalue->{$CurrentColumnName} = $self-
+>default->[$_];
                    }
                } else {
                    # Remove empty field
                    delete $hr_returnvalue->{$CurrentColumnName};
                }
            } 
        }

[download]
but the idea is sound. Devel::Size now reports the returned data-structure to be 385251506 bytes, which, for some reason is smaller than the data-structure retrieved from the db...I'll have to look at things more closely to figure out why that is.

Replies are listed 'Best First'.
Re^3: Reading (the same) data in different ways & memory usage
by BrowserUk (Patriarch) on Apr 19, 2011 at 15:30 UTC
    the returned data-structure to be 385251506 bytes, which, for some reason is smaller than the data-structure retrieved from the db

    Perhaps the DBI code doesn't trim leading/trailing spaces on string fields?

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

      If the target fields in the database are of type CHAR (not VARCHAR), you can make DBI do so by:

      $dbh->{ChopBlanks} = 1;

      Some DBD's do extend this behavior to VARCHAR fields (when the database it too stupid to do so itself, as the ANSI standard tells it to) or when VARCHAR effectively is a CHAR internally (because the database doesn't support VARCHAR).

      YMMV


      Enjoy, Have FUN! H.Merijn
[reply]
[d/l]
[select]
        (when the database it too stupid to do so itself, as the ANSI standard tells it to)

        Erm. If I store the string 'fred          ' into a DB, I'd expect to get that same string back. Does ANSI mandate otherwise?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      Well, one thing I found is that all FLOAT (and probably DOUBLE) values are returned as string (for example "0.00" for a column defined as FLOAT(7,2)).
      Using DBD::MySQL 4.0.18 (should that matter) with default connection options.
[reply]
        one thing I found is that all FLOAT (and probably DOUBLE) values are returned as string

        That does make a certain amount of sense.

        If you converted floats to their binary numeric representation, then you could introduce typical floating point representation errors. If you are going to do math with them in your perl code, then that will happen anyway and you'd have to deal with it.

        But if you are only going to display them, or store them into another table somewhere, then returning them as strings means that you don't have to deal with those errors.

        Whether that is the reasoning behind it I cannot say, but it could be.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom