The modfl() found means Configure did locate a library function modfl; the prototype NOT found means it didn't find a prototype for it the include file math.h.

There's a comment in Configure:

/* Sigh. many current glibcs provide the function, but do not prototype it. */

but presumably this isn't glibc :)

Anyway, in the absence of a prototype in math.h, Configure (and later, perl) will use a prototype of:

long double modfl (long double, long double *);

but this doesn't seem to be what your library is expecting.

I would recommend trying it without the libswanted change.