Here's a command-line version of bart's excellent solution:

src]$ mkdir empty; cd empty; touch {a,b,c}.{a,b,c}
empty]$ echo foo > d.d
empty]$ ls
a.a  a.b  a.c  b.a  b.b  b.c  c.a  c.b  c.c  d.d
empty]$ perl -e'-f and !-s _ and unlink for @ARGV' *.*
empty]$ ls
d.d
empty]$
[download]

Slightly golfier is good on the command line; this is marginally shorter and calls unlink only once, on a list,

empty]$ perl -e'unlink grep -f && !-s _, @ARGV' *.*
[download]

If you don't have to worry about named pipes and other filesystem exotica, you can reduce that to,

empty]$ perl -e'unlink grep !-s, @ARGV' *.*
[download]

since directories are at least one block long.