but isn't this kinda defeating a purpose of abs_path()?

I think you've misunderstood the purpose of abs_path(). It resolves any symlinks along the given path and returns the canonicalized "absolute" path with all symlinks removed (see also "man realpath").

If you give it a relative path, it assumes it's relative to the CWD (as usual), not to whatever readdir() is currently processing...