root@boiao:~# perlbrew list
* /usr/local/bin/perl (5.10.1)
  /usr/bin/perl (5.12.4)
[download]

Those aren't Perls that perlbrew has control over. I found this confusing the first time I used it. Once you install a Perl with perlbrew and switch to it, you see'll the list change. Something like this:

* perl-5.12.4
  /usr/local/bin/perl (5.10.1)
  /usr/bin/perl (5.12.4)
[download]

When you do the installation, you're probably going to want a little more than

perlbrew install perl-5.12.4
[download]

perlbrew (CPAN) documents the general and install-specific options. Also take a look at the two config_args='...' in your perl -V outputs: they differ widely. The perl 5.12.4 INSTALL file may be useful in deciding which -D options to use. You can give each installation its own name with the --as option. So, your actual command might look a bit more like:

perlbrew install -v perl-5.12.4 -Dusethreads -Duselargefiles --as perl
+-5.12.4_custom
[download]

You can now switch to the Perl you've installed. (More on this below.)

If you didn't choose a custom name, your list will now look like what I showed above. If you did use --as, you'll now see

* perl-5.12.4_custom
  /usr/local/bin/perl (5.10.1)
  /usr/bin/perl (5.12.4)
[download]

I noticed you were trying many variations for the switch statement argument. Given the above, the correct incantations would be:

perlbrew switch perl-5.12.4         # --as not used
perlbrew switch perl-5.12.4_custom  # --as used
[download]

You may find this thread useful after installation: perlbrew: moving to a new perl with all current modules