removing the double-quotes makes it run [...] Not sure why

Simple: The first argument of the start command, when put in double quotes, is taken as the requested window title. If you need to quote the command to be started, you also need to provide a dummy window title, i.e.:

system('start "" "C:\\program files\\my application\\kaboom.exe" /foo 
+/bar');
[download]

Also note that system() has a special case on Windows to run a program (with arguments) without waiting for it, and without having to mess with any shell or the braindead start program. This is documented in perlport:

system(1,'C:\\program files\\my application\\kaboom.exe','/foo','/bar'
+);
#      ^-- the 1 is the special case for Windows
[download]

Alexander