IMNSHO the Teradata database is the way to go if you can afford it. It is a proprietary relational DB (with a nice DBI/DBD for Perl). It is a MPP system with multiple nodes that all run in parallel to service each request. It has loading utilities that can load data files such as you describe very quickly. Probably way over your budget though...