in reply to DBI vs Bulk Loading
how much of your process is updating records vs inserting new records?
if it's mostly updates:
select out all the data into your perl program
sum logfiles against data structure in memory by timestamp key
write summed values to csv file
delete table
bulk load csv file
if it's mostly inserting new records:
select out all the data into your perl program
sum logfiles against in memory data structure by timestamp key
run updates for timestamps that exist
write summed values to csv file for new timestamps
bulk load csv file
i dont kow what kind of volume you are processing, but if the full table is too much for your process to handle, you can slice it up by day/hour.
-HTH