Oh yes, CVS can provide no end to entertaining weirdness. My first use (and first big mistake) with CVS is documented for the world to see here on SourceForge. I managed to accidently upload a chunk of my home directory (including GnuPG keys) before I realized what I had done and shut it off. I then got my first lesson in how well CVS supports deleting files.

CVS lets you delete but you still have a copy in the Attic.

But it's a big pain to do the deletion, especially when you have to do it one file at a time and have to put in your SSH password each time (should have gotten my SSH key uploaded to SourceForge, but I was new to SSH at the time, too). And, as you mentioned, it doesn't really delete the file, unless you can get into the guts of the CVS database (which for SourceForge, means taking out a support ticket and waiting). So my old GnuPG secret key ring is there for the world to download (though the first thing I did was revoke the keys).

----
: () { :|:& };:

Note: All code is untested, unless otherwise stated