First off, I'd suggest getting rid of the system/open/while/close code. Use glob and sort instead. Then there also won't be any \n's.

Then, instead of popping 5 times, just splice off the end.

Finally, I'm not sure about File::Remove, but File::Path also has a rmpath function which would do the same trick.

Debugging trick: don't actually remove anything in your code until you've proven it to work. Use print "Pseudo-deleting $tree"; first. Make sure the five you don't want there aren't there.