Artifact Page

Investigating Next Steps in Static API-Misuse Detection

Implementation of MUDetect

The source code of our detector MUDetect is available on GitHub.

Extension of MUBench

The dataset by which we extended the original MUBench benchmark is available as the MUBench experiment dataset FSE18-Extension. To run experiments on the extension, start MUBench with --dataset FSE18-Extension.

Evaluation Results

Experiment RNK

• Full version of Table II: Performance of Each Individual Ranking Strategy

Experiment P

• Statistics for sampling the additional target projects from our MUBench dataset extension
• Assessment of top-20 detector findings, including root-cause classification for MUDetect
• Summary of the results
• Full list of false-positive root causes (with description)
• Reviewer agreement statistics (Kappa Scores)

Experiment RUB

• Assessment of detectors' potential hits, including root-cause classification for MUDetect
• Summary of the results
• Full list of false-negative root causes (with description)
• Reviewer agreement statistics (Kappa Scores)

Experiment R

• Assessment of detectors' potential hits
• Number of training examples per target misuse (column 'In Project')
• Summary of the results
• Reviewer agreement statistics (Kappa Scores)

Experiment XP

• Example BOA query for obtaining usage examples for an API (java.util.List)
• List of sample projects per API (yml format)
• Number of sample projects/training examples per API
• Assessment of top-20 findings of MUDetectXP for measuring its precision (together with Experiment P results)
• Assessment of potential hits of MUDetectXP for measuring its recall (together with Experiment R results)
• Number of training examples per target misuse (column 'X Projects')
• Summary of the results
• Reviewer agreement statistics (Kappa Scores)

Experiment Review Site (MUBench)

We publish the MUBench Review Site with all the experiment data. Please note that this review site does not show the reviews for the findings of the detectors Jadet, GrouMiner, Tikanga, and DMMC on the original MUBench dataset, because these were reviewed in the experiments for "A Systematic Evaluation of Static API-Misuse Detectors" and the results are publicly available on http://www.st.informatik.tu-darmstadt.de/artifacts/mubench/reviews/tse17/.

Pull Requests for Previously-Unknown Misuses

We publish a list of the Pull Request we created from the previously unknown misuses of MUDetect and MUDetectXP.

Experiment Reproduction Pack (MUBench)

Data

We provide the full experiment data (a tarball of the MUBench instance used to run the experiment (~125GB), including all target-project checkouts, detector binaries, and detector findings lists). Note that we had to split the tarball (71.98GB) to a multi-part ZIP (7GB parts) due to upload restrictions of archive.org:

1. mubench-md-study-data.zip
2. mubench-md-study-data.z01
3. mubench-md-study-data.z02
4. mubench-md-study-data.z03
5. mubench-md-study-data.z04
6. mubench-md-study-data.z05
7. mubench-md-study-data.z06
8. mubench-md-study-data.z07
9. mubench-md-study-data.z08
10. mubench-md-study-data.z09

To use the data, install MUBench, mount the checkouts, checkout-xp, detectors, and findings directories from the tarball into the MUBench docker container, and rerun experiments by executing the respective commands listed below. Afterwards, you may publish the detector findings to a review site for manual assessment (./mubench reviewsite start).

Experiment P (Precision)

./mubench run ex2 DMMC --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G
./mubench run ex2 GrouMiner --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G
./mubench run ex2 Jadet --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex2 Tikanga --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex2 MUDetect --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G
./mubench run ex2 MUDetectXP --tag FSE18 --with-xp --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G

Experiment RUB (Recall Upper Bound)

./mubench run ex1 DMMC --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --java-options Xmx16G
./mubench run ex1 GrouMiner --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --java-options Xmx16G
./mubench run ex1 Jadet --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex1 Tikanga --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex1 MUDetect --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --timeout 7200 --java-options Xmx16G

Experiment R (Recall)

./mubench run ex3 DMMC --tag TSE17 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G
./mubench run ex3 GrouMiner --tag TSE17 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G
./mubench run ex3 Jadet --tag TSE17 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex3 Tikanga --tag TSE17 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex3 MUDetect --tag FSE18 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G
./mubench run ex3 MUDetectXP --tag FSE18 --with-xp --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G