Interesting, so I set up cpan as per http://wiki.cpantesters.org/wiki/QuickStart, and did test File::Repl, (I actually ran this a couple of times to play around with it). Here's an extract of what happened, firstly, Win32::API is unsurprisingly unsupported under Linux :)

cpan[1]> test File::Repl
...some stuff redacted...

Configuring D/DR/DROBERTS/File-Repl-2.3.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Warning: prerequisite Win32::API 0 not found.
Generating a Unix-style Makefile
Writing Makefile for File::Repl
Writing MYMETA.yml and MYMETA.json
(/home/marto/perl5/perlbrew/perls/perl-5.26.1/bin/perl Makefile.PL exi
+ted with 0)
CPAN::Reporter: Makefile.PL result is 'pass', No errors.
  DROBERTS/File-Repl-2.3.tar.gz
  /home/marto/perl5/perlbrew/perls/perl-5.26.1/bin/perl Makefile.PL --
+ OK
Running make for D/DR/DROBERTS/File-Repl-2.3.tar.gz
---- Unsatisfied dependencies detected during ----
----       DROBERTS/File-Repl-2.3.tar.gz      ----
    Win32::API [requires]
Running test for module 'Win32::API'
Checksum for /home/marto/.cpan/sources/authors/id/B/BU/BULKDD/Win32/Wi
+n32-API-0.84.tar.gz ok
Configuring B/BU/BULKDD/Win32/Win32-API-0.84.tar.gz with Makefile.PL
OS unsupported
(/home/marto/perl5/perlbrew/perls/perl-5.26.1/bin/perl Makefile.PL exi
+ted with 65280)
CPAN::Reporter: Makefile.PL result is 'na', This platform is not suppo
+rted.
CPAN::Reporter: preparing a CPAN Testers report for Win32-API-0.84
Do you want to review or edit the test report? (yes/no) [no] no
Do you want to send the report? (yes/no) [no] no
[download]

Obviously I didn't want to pollute the test matrix for this module, moving on....

Running make for D/DR/DROBERTS/File-Repl-2.3.tar.gz
Warning: Prerequisite 'Win32::API => 0' for 'DROBERTS/File-Repl-2.3.ta
+r.gz' failed when processing 'BULKDD/Win32/Win32-API-0.84.tar.gz' wit
+h 'writemakefile => NO '/home/marto/perl5/perlbrew/perls/perl-5.26.1/
+bin/perl Makefile.PL' returned status 65280'. Continuing, but chances
+ to succeed are limited.

... lots of stuff redacted ...

all tests completed OK
(/usr/bin/make test exited with 0)
CPAN::Reporter: Test result is 'pass', 'make test' no errors.
CPAN::Reporter: preparing a CPAN Testers report for File-Repl-2.3

CPAN::Reporter: this appears to be a duplicate report for the test pha
+se:
PASS File-Repl-2.3 x86_64-linux 4.5.7-std-3

Test report will not be sent.

  DROBERTS/File-Repl-2.3.tar.gz
Tests succeeded but one dependency not OK (Win32::API)
  DROBERTS/File-Repl-2.3.tar.gz
  [dependencies] -- NA
Failed during this command:
 BULKDD/Win32/Win32-API-0.84.tar.gz           : writemakefile NO '/hom
+e/marto/perl5/perlbrew/perls/perl-5.26.1/bin/perl Makefile.PL' return
+ed status 65280
 DROBERTS/File-Repl-2.3.tar.gz                : make_test NO one depen
+dency not OK (Win32::API)
[download]

Ignoring the fact that I'd been playing around with this, each run CPAN::Reporter marks File::Repl as a 'Pass', since it's test suite passed. The test suite won't drag in Win32::API (https://metacpan.org/source/DROBERTS/File-Repl-2.3/Repl.pm#L157). However, actually running an install for the module fails, as expected.

So based purely on the bare minimum config as per the wiki linked above, it gets a 'pass'. Delving deeper into the cpantesters/smoking config is on my to do list.

I've made minimal changes to the Makefile.PL to get it working. If you have time could you please test it on Windows and let me know if it works? I'll submit a patch.