Sometimes it is hard to beat several decades of development... The classical Unix command line tools written in 'C' should be quite fast in exexuction and development:

sort --merge --unique fileA fileB > output
[download]

If disk space is your major concern, I'd instead compress the input and output files. Text files typically compress to 10% of their initial size. Your command line could then be

zcat fileA.gz fileB.gz | sort --merge --uniq --compress-prog= gzip | g
+zip > output.gz
[download]

Both commands need some disk space in /tmp or in --temporary-directory. RAM usage can be fine tuned with --bufer-size. You'll find out about the --key option by yourself.