Statistics: Results

Experiment 1

Agreements Disagreements Kappa
Detector Projects Synthetics Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 26 3 122 62 0 0 13 49 62 0 0 0 1 0.166 1 13 10.7%
GrouMiner 26 3 122 88 0 0 56 29 85 1 2 3 0.966 0.21 0.957 57 46.7%
Jadet 26 3 122 23 0 0 14 9 23 0 0 0 1 0.238 1 14 11.5%
MuDetect 55 4 171 149 0 0 119 25 144 1 4 5 0.966 0.134 0.961 123 71.9%
Tikanga 26 3 122 15 0 0 3 10 13 0 2 2 0.867 0.133 0.846 3 2.5%
Total 55 4 171 337 0 0 205 122 327 2 8 10 0.97 0.22 0.962 210 24.6%

Experiment 2

:
Agreements Disagreements Kappa
Detector Projects Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Precision
DMMC 5 80 40 2 1 53 54 1 0 1 0.675 0.008 0.672 1 1.3%
GrouMiner 5 90 50 0 0 61 61 1 0 1 0.678 0 0.678 1 1.1%
Jadet 5 52 0 0 2 47 49 2 1 3 0.942 0.036 0.94 4 7.7%
MuDetect 10 146 0 0 29 112 141 2 3 5 0.966 0.153 0.96 32 21.9%
MuDetectXP 10 91 26 3 24 45 69 0 5 5 0.758 0.13 0.722 25 27.5%
Tikanga 5 41 0 0 2 33 35 0 6 6 0.854 0.039 0.848 2 4.9%
Total 10 500 116 5 58 351 409 6 15 21 0.818 0.082 0.802 65 13%

Experiment 3

Agreements Disagreements Kappa
Detector Projects Misuses Potential Hits Open Reviews Open '?' Yes No Total Yes/No No/Yes Total p0 pe Score Hits Recall
DMMC 30 179 42 0 0 13 27 40 0 2 2 0.952 0.199 0.941 13 7.3%
GrouMiner 29 175 14 0 0 7 7 14 0 0 0 1 0.25 1 7 4%
Jadet 29 175 12 0 0 11 0 11 1 0 1 0.917 0 0.917 12 6.9%
MuDetect 56 226 73 0 0 47 26 73 0 0 0 1 0.229 1 47 20.8%
MuDetectXP 51 217 152 0 0 92 55 147 2 3 5 0.967 0.219 0.958 95 43.8%
Tikanga 30 179 11 0 0 8 1 9 2 0 2 0.818 0.066 0.805 10 5.6%
Total 56 226 304 0 0 178 116 294 5 5 10 0.967 0.224 0.958 184 13.6%