Try looking at the CPAN module that connects to a Festival server.
http://search.cpan.org/~rcaley/speech_pm_1.0/Speech/Festival.pm
If you are still interesting in writing your own you can look into how it was implemented to see were you might be going wrong.