536711168 is pretty close to 512M, which (IIRC) is some default memory resource limit on FreeBSD.

What do ulimit -a and limits show?