I think the real question is why you would want to install ImageMagick without X11. Were you having trouble getting it to install? Download the source from
ImageMagick and read the README.txt and Platforms.txt. According to README, "ImageMagick requires an X server for the 'display', 'animate', and 'import' commands to work properly. Next, in the Platforms.txt, they give an example of ImageMagick not working on FreeBSD. It lists two workarounds: one, reboot, which will automatically run ldconfig; or two, after you run make install, run /sbin/ldconfig -m /usr/local/lib (or path to your libs). It's absolutely necessary. Then try reinstalling Image::Magick. Hang in there.
Update: To clear up any confusion, the README listed "reboot"--I've never tried that, and I don't think it would help. But I would recommend running ldconfig.