It would be unusual for a small read size to boost performance, at least with spinning rust. Even if it were the case, changing Perl's minimum read size will almost certainly not make any difference to how the OS buffers file I/O so it's likely that you can only hurt performance by increasing the number of calls into the OS to get work done.

Maybe there is some other reason you are getting a performance hit (assuming this isn't just "I wonder what would happen ..." territory). If you have a real performance problem to solve, perhaps you would like to tell us more about it so we may be able to offer some more substantial advice?