The shebang (or bang) line is not specific to perl. In, say, Bourne shell scripts you see this on the first line:

#!/bin/sh
[download]

See this article for a quick description. I imagine that a lengthier description would require delving into the history of shells and interpreters, but there is no profound significance to the "!".

The term "bang" comes from equating the exclamation symbol "!" with a loud noise. "shebang" is American slang; it's used in the expression "the whole shebang", meaning "the whole thing". As far as I know its meaning has nothing to do with interpreters; it just sounds similar to "bang", so it got adopted by extension, I imagine.