Re: Database input speed question

The simple answer is, don't use DBI. Every decent database has a specialized "loader" program, that bypasses SQL completely and pumps in the raw data for each table from a data file. The various loader programs vary in syntax and features, but for Oracle, have a look at sqlloader, Sybase and MSSQL have a similar program.

The problem is now that you can't have computed fields in your database, as the loader will most likely not have the capabilities, but you can always load the database first and then update the fields that need (Perl) computation.

perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web
[download]

Comment on Re: Database input speed question Download Code

Replies are listed 'Best First'.
Re: Re: Database input speed question by CountZero (Bishop) on Jul 31, 2003 at 19:29 UTC
The loader may not have the intelligence to do computed fields, but Perl does. As most likely you will have to write a Perl script anyhow to reformat your data into a format acceptable to the bulkloader, you can take that opportunity to fill the computed fields. CountZero "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law	[reply]