If you're leaning towards using a RDBMS, I would use an RDBMS. Requirements change and it sounds like you're at the limits of what you consider acceptable performance from flatfiles. Take the time now and convert it to using DBI. Once you have it using DBI, you can access (nearly) any (R)DBMS known to man. (You can use DBD::Sybase to talk to MS-SQL, as well as Win32::ODBC. I would recommend using DBD::Sybase as it allows portability to non-Win32 systems.)

------
We are the carpenters and bricklayers of the Information Age.

The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.