I concur with Mario, benchmarking in any virtualised environment is an almost meaningless act; the hypervisor will allocate your instance as much resource as it uses commensurate with: a) being able to allocate sufficient to all the other instances sharing the physical cpus; b) within the limits of the instance IO restrictions.

And remember, all cloud providers will over-provision physical cpus many times. And it gets worse, sometimes, if a "new" physical CPU has been fired up to run your newly started instance, then you may find yourself the only occupier for a brief while and get performance 10 or 20 times better than a few seconds or minutes later, once the grand overseer has noticed that there is an under utilised box running and transfers in a dozen other instances to soak up the spare capacity.

The only way I've found is to benchmark on standalone hardware for minimal resource -- cpu, io and resource -- not time; and the only meaningful measure of success is the billed cost of running the same job (on the same instance type) before and after optimisation.

And the calculations aren't easy, because the only real control you have is the size of the instance you purchase. There is no point in purchasing a huge compute instance, if your code's performance is bound by the IO limit for that instance. You are just paying more the same throughput.

And your purchase is made in terms of EC2 compute units, and Amazon isn't going to tell you how they are calculated.

Finally, MCE trades IO and CPU for time. On (your own) physical hardware that is a perfectly sane and sensible trade-off. IO costs you nothing, a 90% idle CPU is a waste. Your time is the only factor to consider.

But AWS instances bill you for IO and CPU and time. If for example, using MCE pushed your job into needing a bigger instance that allows higher IO, that could double or treble your costs! It is often better to run 10 tiny instances than one XL instance; if your task lends itself to that.