One observation: You calculate the checksums over both directories in advance. You could save much time if you only calculate it when required.
Meaning:
only if the same filename is in dir1 and dir2, calculate and compare both checksums
you may perform additional (fast) checks before, e.g. only calculate the checksum if the filesize of both files is the same