There may be syntax typos.

For sure. You need to put backslash in front of the pipe symbol when you intend to use it as a literal "vertical bar" character (or you could put quotes around it):

cut -f3 -d\|  X | sort > outx.txt
cut -f3 -d'|' Y | sort > outy.txt
[download]

Provided that part is done right, I prefer using "cut" rather than "awk" -- it's just easier (and probably faster).