This way one can easily replay ↺ a job to retry a failed test and still have the results reported back to Bodhi.
This also means we don't need to hold open the bodhi-trigger job while the test-override jobs runs, which means more slots are available in jenkins to run other tests.
Should resolve https://github.com/coreos/coreos-ci/issues/64
This way we won't get a huge list of RPMs in the messages when people submit large updates. We'll only get listed out the RPMs that we are tracking in CoreOS.