Without bothering to write a benchmark, my guess is that the fastest way is to write your config file as a Perl module - loading the data into variables directly. This means all the parsing is done by C, not Perl. And to be really, really fast, you use just a single scalar variable which stores all the configuration information.

Of course, this has many disadvantages, but you're saying you want the fastest, and you haven't put down any other requirements.