Well, what error message do you receive? According to the documentation, near the New more Tcl/Expect-like interface of $object->expect(), you can return 'exp_continue' symbol from your callback sub to continue matching with next callback.
Are you sure it won't be easier to use some SSH-related modules, for example, Net::SSH::Expect?