Correct. As section 1 of the GPL says:

1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program.

If you do not do this then you are distributing software without the permission of the copyright owner and are in violation of the GPL.

Or at least that is the intent. Whether that will work in practice is open to question. There is, of course, no case law on the GPL. However there does not seem to be much of a question about its enforcability - were there a significant loophole some corporation's legal eagles would have decided not to blink when confronted.

For more on this Eben Moglen has written an essay on Enforcing the GNU GPL. It is worth the read.