alanraetz:

Many people will grab the tool they know best for the job, whether or not it's the right one. I don't know what your expertise is, nor do I know enough about your problem to know if that applies here. But are you sure that perl is the best tool for the task at hand? If the data validations are simple, you may be better served doing the validations in SQL and letting the database do all the heavy lifting. It may be a good deal faster and (ultimately) simpler.

I only mention it, because at work it's all too common to see someone do the equivalent of something like:

$SQL='update tbl set val=? where id=?';
my $ST = $DB->prepare($SQL);
$SQL='select id, val from tbl';
$ar = $DB->selectall_arrayref($SQL);
for my $r (@$ar) {
   if ($r->[1] > 100) {$ST->execute(100, $r->[0])}
   elsif ($r->[1]<0)  {$ST->execute(0, $r->[0])}
}
[download]

This code has two serious problems: First, it extracts *all* rows from the database, when it should extract only the rows needed. Every row fetched from the database consumes time and bandwidth, so you can save a lot of resources by *not* pulling out stuff you don't need. Second, the validations are too simple to need to do the work in perl. It should let the database do the work, something like:

$DB->do("update tbl set val=100 where val>100");
$DB->do("update tbl set val=0 where val<0");
[download]

Any time you need to iterate over a large quantity of data in a database, you should consider whether it's a task that should be done entirely in the database.

Even if the validation is complex enough to dictate some perl code involvement, you may be able to preprocess some of it in SQL to reduce the amount of data you need to read and write to the database.

...roboticus

When your only tool is a hammer, all problems look like your thumb.