should they be inside the brackets as shown?

No, brackets mean optional, see Optional Arguments Notation ( Backus–Naur Form ), its one of the dumber conventions adopted for help documentation, I prefer actual to include actual examples of usage

Do they need to be separated by commas or spaces or something else? Should they be inside quotes?

If they contain spaces or other weird characters, then quoting should be done (easier than escaping), it all depends on your shell

on windows under cmd.exe this might look like 

perl preprocess2_LDfiles.pl  "i am snpF file" iamgffile "i am path inc
+luding pth file"  pS-population-symbol 


perl preprocess2_LDfiles.pl  "i am snpF file" iamgffile "i am path inc
+luding pth file"  pS-population-symbol    th=.8

[download]
I know nothing of this program, and I'm assuming that (snpF) means mandatory argument (href:/http://ki.se/ki/jsp/polopoly.jsp?d=26072&l=en explains that [th=.8] is optional argument)

More tutorials/explanation/examples on shells

See also

In reply to Re: Beginner using a downloaded script (argument notation shell invocation) by Anonymous Monk
in thread Beginner using a downloaded script by TattieSoup

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.