If I lock a table, and another program tries to access it, what happens? ... Does it just block until it becomes unlocked, or is it supposed to return a failure?

This very much depends on what sort of lock you take on a table. If you take a READ lock, then everything else will still be able to read the table also, but any updates to it will be stalled until you release your lock.

If you take a WRITE lock, however, then nothing can read or write the table(s) concerned until you release your lock.

All locked queries will just stall, waiting for the lock to release, rather than failing.

Tony