Start with Wotsit, which has specifications for the most popular graphics formats. You'll want to be familiar with pack and unpack. Hopefully you can read C code.

From there, it's probably a good idea to buy or borrow several graphics algorithms books. Abrash is a good place to start, he'll refer to papers and other theory books you can use to brush up on your math. Hopefully, you're already familiar with linear algebra.

Good luck.