Artifact Page

Investigating Next Steps in Static API-Misuse Detection

Implementation of MUDetect

The source code of our detector MUDetect is available on GitHub.

Extension of MUBench

The dataset by which we extended the original MUBench benchmark is available as the MUBench experiment dataset FSE18-Extension. To run experiments on the extension, start MUBench with --dataset FSE18-Extension.

Evaluation Results

Experiment RNK

Experiment P

Experiment RUB

Experiment R

Experiment XP

Experiment Review Site (MUBench)

We will publish all our review data as a MUBench review page, as soon as this paper is accepted. For now, we provide an anonymize offline version of the page in review-site-anonymized.zip as part of this supplementary material.

Please note that this review site does not show the reviews for the findings of the detectors Jadet, GrouMiner, Tikanga, and DMMC on the original MUBench dataset, because these were reviewed in the experiments for "A Systematic Evaluation of Static API-Misuse Detectors" and the results are publicly available on http://www.st.informatik.tu-darmstadt.de/artifacts/mubench/reviews/tse17/.

Pull Requests for Previously-Unknown Misuses

We publish a list of the Pull Request we created from the previously unknown misuses of MUDetect and MUDetectXP.

Experiment Reproduction Pack (MUBench)

Data

We provide the full experiment data (a tarball of the MUBench instance used to run the experiment (~125GB), including all target-project checkouts, detector binaries, and detector findings lists). Note that we had to split the tarball (71.98GB) to a multi-part ZIP (7GB parts) due to upload restrictions of archive.org:

  1. mubench-md-study-data.zip
  2. mubench-md-study-data.z01
  3. mubench-md-study-data.z02
  4. mubench-md-study-data.z03
  5. mubench-md-study-data.z04
  6. mubench-md-study-data.z05
  7. mubench-md-study-data.z06
  8. mubench-md-study-data.z07
  9. mubench-md-study-data.z08
  10. mubench-md-study-data.z09

To use the data, install MUBench, mount the checkouts, checkout-xp, detectors, and findings directories from the tarball into the MUBench docker container, and rerun experiments by executing the respective commands listed below. Afterwards, you may publish the detector findings to a review site for manual assessment (./mubench reviewsite start).

Experiment P (Precision)

./mubench run ex2 DMMC --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G
./mubench run ex2 GrouMiner --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G
./mubench run ex2 Jadet --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex2 Tikanga --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex2 MUDetect --tag FSE18 --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G
./mubench run ex2 MUDetectXP --tag FSE18 --with-xp --datasets TSE17-ExPrecision FSE18-ExPrecision --timeout 7200 --java-options Xmx16G

Experiment RUB (Recall Upper Bound)

./mubench run ex1 DMMC --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --java-options Xmx16G
./mubench run ex1 GrouMiner --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --java-options Xmx16G
./mubench run ex1 Jadet --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex1 Tikanga --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex1 MUDetect --tag FSE18 --datasets TSE17-ExRecallUpperBound FSE18-Extension --timeout 7200 --java-options Xmx16G

Experiment R (Recall)

./mubench run ex3 DMMC --tag TSE17 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G
./mubench run ex3 GrouMiner --tag TSE17 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G
./mubench run ex3 Jadet --tag TSE17 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex3 Tikanga --tag TSE17 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G Dorg.softevo.oumextractor.javajarsdirs=/opt/jdk/jre/lib
./mubench run ex3 MUDetect --tag FSE18 --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G
./mubench run ex3 MUDetectXP --tag FSE18 --with-xp --datasets TSE17-ExRecall TSE17-ExPrecision-TruePositives FSE18-Extension FSE18-ExPrecision-TruePositives --timeout 7200 --java-options Xmx16G