You might find some clues by looking at a pure trig solution to the problem in Rotate a 3D vector. Maybe try to compare outputs for similar inputs, and see where your logic/math error is? One big error to watch out for is degree/rad mixups.