This smells very much like an NP-complete problem. I'd be
very surprised if the problem isn't known in the literature.
I am although a bit confused by the 'least amount of wasted
space' requirement. If you have 4 4x6 and 6 2x9 rectangles,
it's going to take 252 area. If your big rectangle has area
N, and the smaller rectangles fit, you will be wasting
N - 252 space. No matter how you fit them.