I work a fair bit with matrices, but know nothing about speech recognition. I usually think of a "full covariance" matrix as a variance/covariance matrix with the variances on the diagonal. So, why do you have "0's on the diagonal"?

Without more information I tend to suspect indexing/slicing problems, or that something needs to be transposed, when a matrix misbehaves. If you can post some code someone might be able to help more.