so far I'm just assuming that the sampling rate of the audio file is "accurate"

It might be worth verifying that. What audio recording hardware are you using? In fact, there are some particularly bad soundcards out there, like this one, for example, which could approximately account for the drift you're observing... Most cards are significantly better, though (by orders of magnitude), so this may not necessarily be the problem in your case. But how would you know, unless you've actually measured it...?

A relatively easy way to check would be to record the ticking of a reasonably accurate clock (with an analog display), and then verify that the peaks of the recorded waveform stay in sync with what the audio software reports based on the assumed sample rate. (Even cheap quartz wrist watches typically have of drift of less than one second per day, so that should be a sufficiently precise time reference for the purpose at hand...)

Incidentally, when I was working in psycholinguistic/ERP research, we were using similar experimental setups, to present stimuli, and such... And from those experiences I can confirm that sample rate inaccuracy can in fact become a problem in special cases like the one you're describing.  That was more than a decade ago, though. Hardware might actually have improved since then — or maybe not...