It might be simpler, but maybe I'm using a terrible old and slow modem and on top of that, it takes too many hops to reach your server. In that case, I can switch off CSS and get the default submit button, instead of downloading your whatever.gif, what should be a whatever.jpeg or whatever.png to begin with ;-)

And another nice advantage of the CSS approach is that maybe in a month time you like to restyle your website and you have a new whatever image. The bad thing is that you were cheap and chose a webhosting company that only allows you to use FTP (yeah, they are out there ;-) Now you need to download all the files that contain the reference to this image, change them and upload them again. With the CSS approach, you simple alter the CSS file and upload that one.