Here's an idea: Install the same perl version into a local directory, and then get a directory lising from that installation. You can pipe that list of files to an xargs rm running in /usr/local/.

Assuming that the perl installation didn't override any files, that should be far more reliable and convenient than going through the list by hand.