I think strace wouldn't help here (actually this is one of the very few cases where it wouldn't :), as it would only show something like (in case the shebang line is incorrect, that is)

execve("./SCRIPTNAME", ["./SCRIPTNAME"], [/* 31 vars */]) = -1 ENOENT 
+(No such file or directory)
[download]

The actual shebang resolution happens directly in the kernel, which is invisible to strace-ing.

Note that the "bad interpreter" info is added by bash (as a hint to the user) in case the file permissions of the script itself etc. are OK, but the execve still does return with an error.

Thanx for the word of warning. You mention trailing junk chars below, which I was also interested in with cat -vetting the output. If we can see a syscall, that is (example: touch x; echo "#!/nosachfiel" >x; chmod 755; strace -e file ./x # no userspace syscall for the bad file :().

The only cheap and moderately quick Unix trick I can think of would be to e.g.

1. unmount the nfs,
2. remount (to avoid caching),
3. mount --bind (to force the kernel to stat the interpreter path on some remote NFS; place the client binary on the mount source location to make this more realistic),
4. start tcpdump (to catch the kernel red-handed when it is looking for the interpreter on NFS) and then
5. run the script.

At least the NFS/pcap/tcpdump trick (or an alternative using FUSE/strace) would avoid the overhead of systemtap (C compilers&modprobe while working on all kernels regardless of configuration).

Diffing the hexdumps/ls -l, and checking the mount options should be pretty much what the kernel does itself, unless I've missed something interesting.

As looking for the interpreter in the shebang is kernel-internal only w/o return to userspace, the check of the interpreter should also escape notice when using any of the fakeroot stuff, both LD_PRELOAD and the newer ptrace-based ones. Ugh, another loop hole.

Any good ideas for tracing below strace in Linux (or BSD)? Like a simple-to-use-strace-style frontend to linux systemtap or the probes?

Last time I looked even the AIX4 trace-facility was way more 'quick-use-friendly' than systemtap and friends. And I've still to find a tapset to simulate the global-trace-overkill-from-hell (aka AIX trace-facility). And DTRACE isn't that different (though on Solaris proper I'd expect somewhat better integration in mainline kernels & having more documented examples).