Why not do both? Release the application as open source and offer support and/or customizations for paying customers?

And a pretty manual, too... OSS people don't expect good manuals with their software, but paying customers do. In fact, O'Reilly publishes lots of expensive manuals for free software. There's a market for that.