As it turns out, after looking through much of POE code, that I was doing it all wrong. On the server, I was using my own filter, which wasn't doing the right thing, basically I didn't know that a filter is supposed to work on the raw stream. After I removed my own filter, and let both the server and the client use their default (POE::Filter::Line), all worked out fine.