I do want it to affect 'use' too, as well as require/do.

In that case I think you'll have to use B::Hooks::EndOfScope. For dynamic scoping at runtime, I imagine Filter::Inject could be used to inject a piece of code that basically makes a local change to @INC. For either compile or runtime, if you want it to be lexically scoped, I still can't think of a better way than an @INC hook and %^H. Although I also still can't imagine a use case for use with a lexically scoped @INC :-/