in reply to Data Migration Perl style vs. SQL style....

If the transformations are complicated (i.e., require a lot of procedural logic) and the datasets are relatively small, go ahead and use Perl.

But if the transformations are straight forward and the datasets are large, use SQL. The SQL engine knows how do deal with large amounts of data, can do joins, sorts, etc. all very fast even on large amounts of data. I'd use SQL for all that I could and use Perl for any scripting or procedural logic.

And if you have non-Perl folks involved, using SQL tables as staging areas allows the data to be viewed/verified with what ever tools you have, e.g. Crystal, or any iSQL tool.

  • Comment on Re: Data Migration Perl style vs. SQL style....