What you're seeing is the build cache. It's not kept indefinitely. CPAN.pm deletes stuff from it when it starts overflowing whatever size limit you've set for it. (Though I think it defaults to a fairly generous 100MB.)