Yeah, if a FIFO works then stick with that, it's a lot simpler to deal with. The pty/tty thing will work when an application expects to be able to do some specific ioctl calls on the device (such as DosEMU).
I dug up some more info on what I did and it was basically to use DosEMU to run a DOS based BBS software and make it telnettable (by fooling the BBS software into thinking it was talking to a modem when in reality it was talking to a telnet<->"modem" gateway thing).
Anyway, rambling on here... :D