./Configure -Dprefix=$HOME/perl

That should be all you need, you're prepending -d on your and that says use defaults for all answers and I beleive will override anything after or before it.