How do you plan on deciding which files to load then? Will they be defined by name in some sort of configuration?
You should be setting up an actual distribution structure. See Module::Starter. All of your tests go in t/, and you can then create a test directory structure and associated data in for eg: t/data/. Your module should have a method or something that allows a script to tell it what directory to look in for the files (the tests in the t/*.t test files would do this).
Setting up a proper testing infrastructure allows you to very precisely control everything, and easily and quickly add new sample data and new tests going forward while making changes, or particularly when a bug has been found.
Any chance you could show us the code you have so far?