File1 has three columns

Really? Your first line is:

Box of Joe     CA12DE    12345
[download]

Why is that three columns, not five?

For the rest of the post, I will assume your columns are tab separated. (Maybe your data actually does contain tabs, but I cannot see that). Adjust accordingly if columns are defined differently. Either use a hash, or ~~ (untested):

chomp(my @file2 = <F2>);
while (<F1>) {
    print unless @file2 ~~ (split /\t/)[1];
}
[download]

See also the questions "How can I tell whether a certain element is contained in a list or array?", "How can I remove duplicate elements from a list or array?", and "How do I compute the difference of two arrays? How do I compute the intersection of two arrays?" from the perlfaq. (man perlfaq4).