You are here
Failed Failed Error Propagation
Abstract: When testing software, developers can place oracles externally or internally to a method. Given a faulty execution state, i.e., one that differs from the expected one, an oracle might be unable to expose the fault if it is placed at a program point with no access to the incorrect program state or where the program state is no longer corrupted. In such a case, the oracle is subject to failed error propagation. Internal oracles are in principle less subject to failed error propagation than external oracles. However, they are also more difficult to define manually. Hence, a key research question is whether a more intrusive oracle placement is justified by its higher fault detection capability.
We conducted an empirical study to measure failed error propagation on real faults, considering Defect4J, a benchmark of Java programs, of which we used all 6 projects available, 384 real bugs and 528 methods fixed to correct such bugs. Our results indicate that the prevalence of failed error propagation is negligible. Hence, internal oracle placement is not justified by higher fault detection. Moreover, our results on real faults differ from our results on mutants, indicating that if failed error propagation is taken into account, mutants are not a good surrogate of real faults.