Would it be feasible to give a programming question, ask the test taker to code a solution, and then execute that code with the Safe module and check the output?

One thing you don't want to do as a test administrator is look over every answer. And the test has to be able to take vastly different coding styles into account. The system will need to offer a kind of black box test. Here's your input, I need that output, fill in the box.

oakbox