You're absolutely right about the anchoring, I completely overlooked that.
On some lines, my regex was matching the last letter or the last couple of letters of the first cell instead of the whole cell.
s/^([^\t]*)\t... instead of s/([^\t]*)\t... fixes the problem.