Sereal may be worth looking at. It doesn't have the hooks you're looking for, but based on my experience (porting Sereal::Encoder to PHP) it's relatively clean code if you want to extend it. (Albeit mostly XS.)

It's also an actively developed framework and format, so you might have a good chance of getting your extensions adopted into the Sereal::{Encoder,Decoder} modules.