I can watch the list grow by looking at /proc/PID/fd

I'm not sure I understand. How can you look at /proc/PID/fd, if PID is no longer there (as you're implicitly saying elsewhere)?

Also, what processes does lsof say those file handles belong to? What does close return?