It ignores them in "t/" but just in case you ever need to get CPAN to ignore some modules elsewhere, you can add a "noindex" directive to your META.yml file. For example, to ignore .pm files in an "examples/" directory:

noindex:
  directory:
    - examples
[download]

Read the full META.yml spec for details.

With Module::Build, you just have to add the following to your Build.PL file:

meta_add => {
    noindex => { directory => [ qw/examples/ ] }
},
[download]