This is really a shell question, not a Perl question. Usually shell commands send error messages to stderr; in my experience "--" isn't a file handle, but a flag indicating that there are no more arguments to process.

You've got a couple choices here: the easiest is use the shell's redirection operators within the command passed to system. Others would be to modify the program you're calling via system to put stderr in a specific file, or to figure out how to connect the output to stderr to a Perl filehandle. I don't even know if that's possible.