I can offer several thoughts from my own practices -- no claim that any is right (clearly I don't even have a really consistent standard), but one or more of them may resonate for you.

• For a really simple module, all the tests go in one file in the t/ directory
• For utility modules, I tend to break out test scripts somewhat ad hoc to test different types of functionality. (You might think of this as testing 'use cases')
• For a distribution with lots of classes/subclasses, I've generally gone with one test file per module, testing whatever is unique/different about that module
• In one case, with a really complicated object hierarchy, I've set up subdirectories within t/ to mimic my object hierarchy and customized the Build.PL to search t/ recursively.
• If I find myself with a lot of redundant setup code, I create a "TestHelper.pm" module in the t/ directory and have my test scripts load that. (Though I recently learned about Test::Class and think it might be a better approach for that kind of thing.)

Generally, I consider test scripts cheap and don't hesistate to create a new one to test a narrow set of functionality. That said, I try to keep related tests grouped together in one file.

Also, in your core toolkit of Test modules, make sure you look at Test::Exception. I use it so frequently that my default .t template automatically sticks that in along with Test::More. (And if you're doing a lot of floating-point math, you might appreciate my own Test::Number::Delta, too.)