jdtoronto has asked for the wisdom of the Perl Monks concerning the following question:

Most repsected monks,

Using DBI and MySQL on W32 in an app using Tk.

I have two arrays:

  • @fields which has in it a number of fields read from a flat text file database
  • @map_fields which is the names of the fields in the database that the field data is to be stored in.
  • The two arrays are returned by a sub that allows the user to select an available field from the database for each field of the import.

    The number of fields in the import can vary! The input db's can have as few as 5 or as many as 30 fields and the order of the fields varies also.

    In the past I wrote a thing like this for one particular flat-file layout, I parsed the fields directly into a hash using split:

    sub Read_leads { open (INPUT, "<$lo_in") || die "Cannot open source file $!\n"; while (<INPUT>) { $in_ln = $_; $in_ln =~ s/^\s+//; $in_ln =~ s/\s+$//; ($lead{Email}, $lead{FirstName}, $lead{LastName} , $lead{DOB}, + $lead{Gender}, $lead{Country}, $lead{State}, $lead{ZIP}, $lead{Addre +ss}, $lead{City}) = split(/,/, $in_ln); $lead{DATE} = $indate; if ($lead{Email}) { $lead{Datasource} = '10222'; $rec = {}; while (($k, $v) = each %lead) { $v = $dbh->quote( $v); $rec->{$k} = $v; } push @leadslist, $rec; } %lead = (); } return; }
    Then after a couple of other simple operations I took the array of hashes and stored it like this:
    sub Write_leads { @fields = qw(Email FirstName LastName Address City State ZIP Datas +ource AdType OrigIP DATE Gender DOB phone Country); $sth = $dbh->prepare("INSERT INTO $leads_tbl VALUES (?,?,?,?,?,?,? +,?,?,?,?,?,?,?,?)"); for $href (@leadslist) { $sth->execute( @{$href}{@fields}); } return; }
    But how big can the array of hashes get? OK, I can do the import line by line and play it safe, thats no problem.

    But how do I handle the variable nature of what I am trying to do now?

    Many thanks in anticipation!

    John

    Replies are listed 'Best First'.
    Re: DBI and two arrays question!
    by dragonchild (Archbishop) on Jul 24, 2003 at 18:41 UTC
      First - use strict; - you will have a million issues if you don't. Secondly, PASS THINGS IN! You're using global variables. This is highly unmaintainable code.

      If I understand your question right, you might be passing @fields in to Write_leads(). Ok ... that's not a problem.

      sub Write_leads { my ($fields, $table, $list) = @_; my $parms = join (',', ('?') x @$fields); my $sth = $dbh->prepare_cached("INSERT INTO $table VALUES ($parms) +") || die $dbh->Errstr; foreach my $href (@$list) { $sth->execute(@{$href}{@$fields}) || die $dbh->Errstr; } $sth->finish; }
      That not only will run under strict, but will catch DB errors you might have (for example, your tablename which is a variable might be wrong ...)

      ------
      We are the carpenters and bricklayers of the Information Age.

      Don't go borrowing trouble. For programmers, this means Worry only about what you need to implement.

      Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

    Re: DBI and two arrays question!
    by tadman (Prior) on Jul 24, 2003 at 18:45 UTC
      An array of hashes can get as big as you want, limited only by available memory and swap. If you're really worried, you could do the import line-by-line, as you suggest, if you're working with hundreds of megabytes of data.
    Re: DBI and two arrays question!
    by graff (Chancellor) on Jul 25, 2003 at 02:07 UTC
      I think you may want to elaborate your insert statement a bit, so that it does not depend on some "hidden" ordering of fields in the database table. If I understood your opening statement, "@map_fields" contains the set of field names to be handled for a given table / set of rows, and "@fields" would store the actual column values for a given row. In that case, something like this would make sense:
      my $fieldnames = join( ",", @map_fields ); my $parmstring = join( ",", ('?') x scalar @map_fields ); my $sth = $dbh->prepare( "INSERT INTO $some_tbl ($fieldnames) VALUES + ($parmstring)" ); # loop over rows of data with the execute call: ... $sth->execute( @fields ); ...
      This way, as long as your input data provides the correct field labels in the correct order to go with each set of actual data rows, you'll be fine -- having the field names, in a specific order, provided in the insert statement will save you a lot of grief.