should they be inside the brackets as shown?

No, brackets mean optional, see Optional Arguments Notation ( Backus–Naur Form ), its one of the dumber conventions adopted for help documentation, I prefer actual to include actual examples of usage

Do they need to be separated by commas or spaces or something else? Should they be inside quotes?

If they contain spaces or other weird characters, then quoting should be done (easier than escaping), it all depends on your shell

on windows under cmd.exe this might look like

perl preprocess2_LDfiles.pl  "i am snpF file" iamgffile "i am path inc
+luding pth file"  pS-population-symbol 


perl preprocess2_LDfiles.pl  "i am snpF file" iamgffile "i am path inc
+luding pth file"  pS-population-symbol    th=.8
[download]

I know nothing of this program, and I'm assuming that (snpF) means mandatory argument (href:/http://ki.se/ki/jsp/polopoly.jsp?d=26072&l=en explains that [th=.8] is optional argument)

More tutorials/explanation/examples on shells