Personally, I don't use the xt directory. All of my tests regardless of the context(s) they run, are all in the t/ directory (in some cases I have multiple levels of directories under t/).

Typically, POD checking tests, MANIFEST tests, critic tests and pretty much any test that checks something that doesn't have any impact on the compilation, installation or operation of your distribution could be classified as an author test.

I use the environment variable RELEASE_TESTING to enable/disable my author tests. I typically only run them when I'm about to tag/upload to CPAN a new release. I usually skip author tests like this:

use strict;
use warnings;
use Test::More;
 
unless ( $ENV{RELEASE_TESTING} ) {
    plan( skip_all => "Author tests not required for installation" );
}
 
my $min_tcm = 0.9;
eval "use Test::CheckManifest $min_tcm";
plan skip_all => "Test::CheckManifest $min_tcm required" if $@;
 
ok_manifest();
[download]

To further, several of my distributions have numerous other tests that skip under certain scenarios during install, as the end-user may not have all of the requisites configured or installed. For example, I have a dedicated hardware setup for continuous integration/testing for my RPi::WiringPi suite with all sensors and ICs attached. Most won't have this on install, so in my documentation for said distribution, I have:

Testing Environment Variable List

Here's the contents of my /etc/environment file, setting the various t
+esting environment variables for the full test platform. For LCD, the
+ last two digits (4, 20) are for four row, 20 column units. If you on
+ly have a two row by 16 column unit, leave those last two digits off.

PI_BOARD=1
RPI_ARDUINO=1
RPI_ADC=1
RPI_MCP3008=1
RPI_MCP4922=1
RPI_SHIFTREG=1
RPI_LCD=1
RPI_SERIAL=1
RPI_HCSR04=1
BB_RPI_LCD=5,6,4,17,27,22,4,20
RPI_RTC=1
RPI_MCP23017=1
RELEASE_TESTING=1
[download]

That tests the entire platform and all connected devices, ICs, sensors etc, including my author tests. None of these are set in a user's environment by default, so the most basic of tests will pass on any platform, skipping the ones that can't be tested.

To skip them, it's very similar to the author tests:

BEGIN {
    if (! $ENV{RPI_ARDUINO}){
        plan skip_all => "RPI_ARDUINO environment variable not set\n";
    }

    if (! $ENV{PI_BOARD}){
        $ENV{NO_BOARD} = 1;
        plan skip_all => "Not on a Pi board\n";
    }
}
[download]

The test output is displayed like the following (some cleanup for brevity's sake) when not running on a Raspberry Pi:

PERL_DL_NONLAZY=1 "/home/pi/perl5/perlbrew/perls/perl-5.28.0/bin/perl"
+ "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harne
+ss::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ....................... ok   
t/01-identification_and_label.t ... skipped: Not on a Pi board
t/05-pin.t ........................ skipped: Not on a Pi board
t/10-register.t ................... skipped: Not on a Pi board
t/15-pwm_spi_adc.t ................ skipped: RPI_ADC environment varia
+ble not set
t/20-cleanup.t .................... skipped: Not on a Pi board
t/25-sig_die.t .................... skipped: Not on a Pi board
t/35-pin_map.t .................... skipped: Not on a Pi board
t/40-interrupt_rising_and_pud.t ... skipped: Not on a Pi board
t/41-interrupt_falling_and_pud.t .. skipped: Not on a Pi board
t/42-interrupt_both_and_pud.t ..... skipped: Not on a Pi board
t/45-shift_reg_adc.t .............. skipped: RPI_SHIFTREG environment 
+variable not set
t/55-dac.t ........................ skipped: RPI_MCP4922 environment v
+ariable not set
t/60-lcd.t ........................ skipped: RPI_LCD environment varia
+ble not set
t/64-i2c_exceptions.t ............. skipped: RPI_ARUDINO environment v
+ariable not set
t/65-i2c.t ........................ skipped: RPI_ARDUINO environment v
+ariable not set
t/67-rtc.t ........................ skipped: RPI_RTC environment varia
+ble not set
t/70-alt_modes.t .................. skipped: Not on a Pi board
t/75-serial.t ..................... skipped: RPI_SERIAL environment va
+riable not set
t/80-mode_state_all_pins.t ........ skipped: Not on a Pi board
t/85-pwm_hw_mods.t ................ skipped: Not on a Pi board
t/90-servo.t ...................... skipped: Not on a Pi board
t/92-mcp23017.t ................... skipped: RPI_MCP23017 environment 
+variable not set
t/95-pod_linkcheck.t .............. skipped: Test::Pod::LinkCheck requ
+ired for testing POD links
t/manifest.t ...................... skipped: Author tests not required
+ for installation
t/pod-coverage.t .................. skipped: Author tests not required
+ for installation
[download]

The third one from the bottom is both an author test file (pod_linkcheck.t), and it has a prerequisite. I don't want to include unnecessary distributions if I don't have to, so on my testing platforms, the prereq is installed, so if RELEASE_TESTING=1 is set, that test file will run. Here's how I check for the distribution's availability:

use strict; use warnings;
  
use Test::More;
  
eval "use Test::Pod::LinkCheck";
if ($@) {
    plan skip_all => 'Test::Pod::LinkCheck required for testing POD';
} 
if (! $ENV{RELEASE_TESTING}){
    plan skip_all => 'Test::POD::LinkCheck tests not required for inst
+all.';
}
 
Test::Pod::LinkCheck->new->all_pod_ok;
[download]

Not the best way to use eval(), but it doesn't matter. Any eval error will suffice here to skip the tests.