I have no experience with PowerShell, but if it has similar interpolation syntax to unix shells, you have to use doublesingle quotes: perl -e '$_ = 1'

In unix shells variables are interpolated into double quotes, and since you probably have no variable $_ defined in your shell, the one-liner that perl sees looks l ike this:  = 1;

Update: s/double/single/, thank you BrowserUk