Basically, you found a bug in the implementation:

BOOL
KillProcess(pid, exitcode)
    DWORD pid
    unsigned int exitcode
CODE:
    {
        HANDLE ph = OpenProcess(PROCESS_ALL_ACCESS, 0, pid);
        if (ph) {
            RETVAL = TerminateProcess(ph, exitcode);
            if (RETVAL)
                CloseHandle(ph);
        }
    }
OUTPUT:
    RETVAL
[download]

Or maybe several bugs.

1. If your perl process is not running with sufficient priviledges to open the target process PROCESS_ALL_ACCESS, then RETVAL is never being (visibly) set and you are getting whatever value it is initialised to by XSUBPP (if any, the docs are less than clear about this), or some random value in the ram it inherits.
2. If OpenProcess() succeeds, then you get the return value of TerminateProcess(), which is non-zero for success, or zero for failure in which case calling GetLastError() should tell you why it failed.

It's possible that when it fails, the arbitrarily large number you are getting is the OS failure code. It would be worth comparing it against the numeric realisation of $^E.