... makes me think maybe we used to trust the cached value
I think https://www.nntp.perl.org/group/perl.perl5.porters/2016/10/msg240576.html and responses is relevant here.
As also mentioned there, the stringification of the NV is retained if the NV is an inf or nan.
I assumed this was ok because stringifications of inf and nan were not affected by locale. (But I notice that stringification of 0.0 to "0" is not retained.)