You can use a while loop around the readdir call, and push the files into your @files, @dirs arrays using a if, elsif, else construct. no need for intermediary @all.
If you don't like this solution for being too simple or verbose, then you could always play with map in void context and the ? : switches.