Is there a VERY fast (maybe implemented in C) way to calculate intersection and union on two arrays?

You may want to have a look at PDL. And, no doubt, the fastest routines have probably been written in the early 60s, in FORTRAN. But I wouldn't know how to link FORTRAN routines to perl.