Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to parse fields from multiple files in a directory and then insert them in mysql database.
For example:
File format:
id=22 name="Mark Henry" age=35 address="82, Linking road, Parkplaza, 56532"
idno=22 name="Jack Henry" age=28 address="10, Sudon village, Brownie, 3251"
Mysql table:
id name age address
my code:
#!perl #use strict; #use warnings; use File::DosGlob 'glob'; use DBI; use DBD::mysql; my $dir = "/user/data/"; my @data; $database = userdata; $dbh = DBI->connect('dbi:mysql:userdata','root','') or die "Connection + Error: $DBI::errstr\n"; chdir $dir; for my $file (grep {-f} glob '2010-05*') { open my $fh, '<', $file; while (<$fh>) { if (my ($id) = /\bid=(\d+)/) { push @ids, $id; } if (my ($name) = /\bname=("(.*?)")/i) { push @names, $name; } if (my ($age) = /\bage=(\d+)/) { push @ages, $age; } if (my ($add) = /\baddress=("(.*?)")/i) { push @adds, $add; } } $ids1 = join(' ',@ids); $name1 = join(' ',@names); $age1 = join('',@ages); $add1 = join('',@adds); $query = "INSERT into `userdata` ( `id`, `name`, `age`, `address`) val +ues ('$id1', '$nam1', '$age1', '$add1')"; $dbh->do($query); }
This script is not inserting all values of corresponding fields in mysql database.
Please help.

Replies are listed 'Best First'.
Re: Inserting data in Mysql
by marto (Cardinal) on Dec 03, 2010 at 10:41 UTC

    Don't comment out use strict; use warnings;. See the DBI documentation regarding placeholders. You're not checking your inserts work:

    my $rows = $dbh->do($statement) or die $dbh->errstr;

    Which is also covered in the documentation. For debugging check that you have a value for each of the fields, for example:

    Warn "value1: $value1, value2: $value2...";
      Or instead of checking for errors, you can use the RaiseError => 1 option in DBI->connect. Very useful, and has all the desired information in the error messages.
Re: Inserting data in Mysql
by roboticus (Chancellor) on Dec 03, 2010 at 11:41 UTC

    A few notes:

    1. Using placeholders is really the correct choice for something like this. It handles all your quoting for you, properly handles nulls, and simplifies your code. A complete win.
    2. Never comment out strict and warnings. If they're telling you something, you ought to first understand what they're telling you and repair the problems rather than mask any problems and plowing ahead. (Yes, there are cases where it's appropriate to turn some of it off, but when you investigate it properly, you'll find out when to do so.)
    3. I don't use MySQL, but I really don't think you can insert multiple records as you're trying to do.
    So I rearranged your code a little, converted to use placeholders and insert the records as they're read. I didn't bother to compile and/or test, so there could easily be a few syntax errors. But this should give you a push in the right direction:

    #!perl use strict; use warnings; use File::DosGlob 'glob'; use DBI; use DBD::mysql; my $dir = "/user/data/"; $database = userdata; $dbh = DBI->connect('dbi:mysql:userdata','root','') or die "Connection Error: $DBI::errstr\n"; my $sth = $dbh->prepare(<<EOSQL) or die "Can't prepare insert statement: $DBI::errstr\n"; INSERT into userdata(id, name, age, address) values (?, ?, ?, ?) EOSQL chdir $dir; for my $file (grep {-f} glob '2010-05*') { open my $fh, '<', $file; while (<$fh>) { my ($id) = /\bid=(\d+)/; next if not defined $id; # Can't store without a key! my ($name) = /\bname=("(.*?)")/i; my ($age) = /\bage=(\d+)/; my ($add) = /\baddress=("(.*?)")/i; $sth->execute($id, $name, $age, $add) or die $DBI::errstr; } }

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

      Pay particular attention to the fact that, in the example above, the question-marks are not enclosed in quotes.   If they were, they’d be seen as a literal character-string consisting of a question mark.   For some strange reason, I whacked my forehead against that quite a few times, in my earlier days.   (Thank you ... I feel so much better now ...)

      INSERT and UPDATE statements are two of the biggest reasons for having placeholders, and they’re also one of their biggest beneficiaries.   When you do these things, you’re inserting user-provided data (which might be malicious), and you’re probably doing it thousands of times, maybe under a fairly tight schedule.   So, prepare the statement-handle once, then keep re-using it.

      If you are doing many inserts, be sure to do it within a transaction.   (COMMIT periodically if it makes sense to do so.)   This tells the database server that it doesn’t have to ensure that the data has been committed to disk before the statement completes.   For an SQLite database-file, for instance, this simple concern makes a “night vs. day” difference in performance.   Naturally, there are auto_commit features available...

Re: Inserting data in Mysql
by JavaFan (Canon) on Dec 03, 2010 at 11:07 UTC
    You're doing a single query per file. For instance, with the given two line example, your query will be:
    INSERT into userdata (id, name, age, address) values ('22', 'Mark Henr +y Jack Henry', '3528', '82, Linking road, Parkplaza, 5653210, Sudon v +illage, Brownie, 3251')
    Assuming you fix the '$nam1' typo. Is that really what you want to do? Note that your id field could easily by '1 2 3 4', which doesn't seem like a very useful id to me.
Re: Inserting data in Mysql
by samarzone (Pilgrim) on Dec 03, 2010 at 11:36 UTC

    Did you notice that second record in your example data starts with "idno=" instead of "id=" while this is not taken care of in the regex in your script?

    Is this a typo?

      Sorry that was a typo
Re: Inserting data in Mysql
by ww (Archbishop) on Dec 03, 2010 at 14:35 UTC
    Your regex,
      if (my ($id) = /\bid=(\d+)/),
    won't match some of your data; to wit:
    idno=22 name="Jack Henry" age=28 address="10, Sudon village, Brownie, 3251"

    Small sidebar re your formatting: please use <c>...</c> tags around your data -- including, for example, not only the data you show us, but also the tables, id name age address