Statistics: Results

Experiment 1

Agreements Disagreements Kappa
Detector Projects Synthetics Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 29 25 64 40 0 0 15 20 35 4 1 5 0.875 0.19 0.846 15 23.4%
GrouMiner 29 25 64 46 0 0 28 12 40 2 4 6 0.87 0.163 0.844 31 48.4%
Jadet 29 25 64 19 0 0 12 3 15 3 1 4 0.789 0.108 0.764 15 23.4%
Tikanga 29 25 64 23 0 0 11 9 20 1 2 3 0.87 0.191 0.839 13 20.3%
Total 29 25 64 128 0 0 66 44 110 10 8 18 0.859 0.182 0.828 74 28.9%

Experiment 2

:
Agreements Disagreements Kappa
Detector Projects Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Precision
DMMC 5 81 0 0 6 68 74 2 5 7 0.914 0.064 0.908 8 9.9%
GrouMiner 5 66 0 0 0 64 64 0 2 2 0.97 0 0.97 0 0%
Jadet 5 39 0 0 3 35 38 1 0 1 0.974 0.069 0.972 4 10.3%
Tikanga 5 44 0 0 4 37 41 0 3 3 0.932 0.076 0.926 5 11.4%
Total 5 230 0 0 13 204 217 3 10 13 0.943 0.051 0.94 17 7.4%

Experiment 3

Agreements Disagreements Kappa
Detector Projects Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 29 53 25 0 0 11 13 24 0 1 1 0.96 0.229 0.948 11 20.8%
GrouMiner 29 53 4 0 0 0 4 4 0 0 0 1 0 1 0 0%
Jadet 29 53 4 0 0 3 1 4 0 0 0 1 0.188 1 3 5.7%
Tikanga 29 53 9 0 0 7 2 9 0 0 0 1 0.173 1 7 13.2%
Total 29 53 42 0 0 21 20 41 0 1 1 0.976 0.238 0.969 21 9.9%