$ENV{'REMOTE_ADDR'} is the address of the remote end of the connection on which the HTTP request was received. In other words, it's the address from which the request came, and it's the address to which the answer will be returned. Assuming you trust your web server, the address can only be spoofed if the user can guess the sequence number the web server used to start the conversation (since it's TCP, and not UDP). That's not easy (which doesn't meant it's not possible), and I've never heard of it being done outside of academics. I read a good article discussing how vulnerable TCP sequence numbers are, but I can't find it.

It doesn't mean it's his IP address, however. He could be using another machine as a proxy or tunnel, or he could be controlling a web client on a remote machine.