Interesting suggestion - but that just feels wrong to me. One class to one file - otherwise debugging becomes bizarre if there's an error. Imagine the confusion parsing X.pm looking for a method that's declared in X::Y.pm uder a package X declaration!
No more wrong than having a method in X return a new X::Y object. So long as you stick to a consistent way to do your factory classes, I don't see a problem.