Have you considered DBD::CSV? It uses Text::CSV_XS under the hood (speed), reads all in memory (database tables) and gives you a SQL interface to your data.

If all you need is to get all CSV into memory as quickly as possible, have a look at Text::CSV_XS' getline_all ()