Nice, I didn't know about CatalystX::Starter and just did one of my own by hand the same way. Where were you when I needed you?!

You don't always need a test application though. Maybe usually don't. They're for testing generic, typically application-free, parts like plugins and base controllers and such. If you're writing a real app, you can test it directly, don't need to mimic it in a stripped down test version.