Maybe the NFS share is mounted as "noexec"? (I think that would produce the exact error you're getting, whereas some trailing junk char(s) in the shebang line would rather produce "... No such file or directory").

What does mount show for the share in question?

YOU WIN! It is indeed mounted as noexec. However, I'm mounting it in fstab, and the options are only 'defaults,user'. I'll google around and see what I'm doing wrong. Cheers for that... and to everyone else that gave their 2 cents worth.

Some ideas to try

• compare the hex dumps (od -x, hd, xxd, ...) of the first few lines. On the file server locally, "copied" version (how did you copy it?) and accessed-via-nfs.
• implicit conversion might also be part of the mount options. What OS is the server, what is the client?
• strace -o log -e file SCRIPTNAME # what files are accessed?
• strace -o log -e file perl ./SCRIPTNAME # (use cat -vet when viewing the log)

cu
Peter

I think strace wouldn't help here (actually this is one of the very few cases where it wouldn't :), as it would only show something like (in case the shebang line is incorrect, that is)

execve("./SCRIPTNAME", ["./SCRIPTNAME"], [/* 31 vars */]) = -1 ENOENT 
+(No such file or directory)
[download]

The actual shebang resolution happens directly in the kernel, which is invisible to strace-ing.

Note that the "bad interpreter" info is added by bash (as a hint to the user) in case the file permissions of the script itself etc. are OK, but the execve still does return with an error.

Thanx for the word of warning. You mention trailing junk chars below, which I was also interested in with cat -vetting the output. If we can see a syscall, that is (example: touch x; echo "#!/nosachfiel" >x; chmod 755; strace -e file ./x # no userspace syscall for the bad file :().

The only cheap and moderately quick Unix trick I can think of would be to e.g.

1. unmount the nfs,
2. remount (to avoid caching),
3. mount --bind (to force the kernel to stat the interpreter path on some remote NFS; place the client binary on the mount source location to make this more realistic),
4. start tcpdump (to catch the kernel red-handed when it is looking for the interpreter on NFS) and then
5. run the script.

At least the NFS/pcap/tcpdump trick (or an alternative using FUSE/strace) would avoid the overhead of systemtap (C compilers&modprobe while working on all kernels regardless of configuration).

Diffing the hexdumps/ls -l, and checking the mount options should be pretty much what the kernel does itself, unless I've missed something interesting.

As looking for the interpreter in the shebang is kernel-internal only w/o return to userspace, the check of the interpreter should also escape notice when using any of the fakeroot stuff, both LD_PRELOAD and the newer ptrace-based ones. Ugh, another loop hole.

Any good ideas for tracing below strace in Linux (or BSD)? Like a simple-to-use-strace-style frontend to linux systemtap or the probes?

Last time I looked even the AIX4 trace-facility was way more 'quick-use-friendly' than systemtap and friends. And I've still to find a tapset to simulate the global-trace-overkill-from-hell (aka AIX trace-facility). And DTRACE isn't that different (though on Solaris proper I'd expect somewhat better integration in mainline kernels & having more documented examples).