Update: Curtis Leach had a similar issue in 2009 and sought help on the internet, with no response to his issue. It is the similar to here. As he pointed out, and I confirmed on my machine, there are four possible combinations here and one of the four fails (unfortunately, it is the critical one for my situation and Curtis').

1. Encrypt with gpg and decrypt with gpg.
   Works fine, as expected
2. Encrypt with perl using Crypt::OpenPGP and decrypt with gpg.
   Works fine.
3. Encrypt with perl using Crypt::OpenPGP and decrypt with perl using Crypt::OpenPGP
   Works fine.
4. Encrypt with gpg and decrypt with perl using Crypt::OpenPGP
   Failure with "Symkey decrypt failed: Invalid secret key ID" message. This is the situation described above

In fact, for the last combination in my case, the file was encrypted by javascript on another machine. However, if I generated an encrypted file with gpg the same error arises.