Statfier
| Statfier is an heuristic-based automated testing tool for static analyzers. Basically, it is implemented by program transformation and metamorphic testing techniques. Until now, Statfier has found 79 bugs in known static analyzers (PMD, SpotBugs, SonarQube, CheckStyle, Infer). |
Defects found
| Tool | Link | Root cause | Status |
| PMD | Link | Variable declaration | Fixed |
| SonarQube | Link | Control flow structure | Fixed |
| SonarQube | Link | Variable declaration | Fixed |
| checkstyle | Link | Compound expression | Fixed |
| SpotBugs | Link | Class hierarchy | Confirmed |
| PMD | Link | Variable declaration | Fixed |
| PMD | Link | Control flow structure | Pending |
| PMD | Link | Compound expression | Confirmed |
| checkstyle | Link | Compound expression | Fixed |
| SpotBugs | Link | Variable declaration | Fixed |
| checkstyle | Link | Class hierarchy | Fixed |
| checkstyle | Link | Class hierarchy | Confirmed |
| PMD | Link | Compound expression | Confirmed |
| PMD | Link | Java version and new features | Pending |
| PMD | Link | Variable declaration | Pending |
| PMD | Link | Variable declaration | Pending |
| PMD | Link | Variable declaration | Fixed |
| PMD | Link | Class hierarchy | Fixed |
| SpotBugs | Link | Java version and new features | Confirmed |
| PMD | Link | Variable declaration | Fixed |
| PMD | Link | Class hierarchy | Confirmed |
| PMD | Link | Class hierarchy | Confirmed |
| PMD | Link | Class hierarchy | Confirmed |
| PMD | Link | Class hierarchy | Confirmed |
| PMD | Link | Class hierarchy | Confirmed |
| PMD | Link | Class hierarchy | Confirmed |
| SpotBugs | Link | Control flow structure | Pending |
| checkstyle | Link | Java version and new features | Fixed |
| checkstyle | Link | Control flow structure | Won't fix |
| PMD | Link | Control flow structure | Fixed |
| SpotBugs | Link | Java version and new features | Confirmed |
| SonarQube | Link | Control flow structure | Won't fix |
| SonarQube | Link | Control flow structure | Confirmed |
| SonarQube | Link | Control flow structure | Confirmed |
| SonarQube | Link | Variable declaration | Won't fix |
| SonarQube | Link | Variable declaration | Confirmed |
| PMD | Link | Variable declaration | Pending |
| PMD | Link | Variable declaration | Confirmed |
| PMD | Link | Compound expression | Pending |
| PMD | Link | Control flow structure | Fixed |
| PMD | Link | Compound expression | Confirmed |
| PMD | Link | Class hierarchy | Pending |
| PMD | Link | Compound expression | Pending |
| PMD | Link | Class hierarchy | Fixed |
| PMD | Link | Variable declaration | Fixed |
| checkstyle | Link | Compound expression | Won't fix |
| checkstyle | Link | Class hierarchy | Pending |
| SpotBugs | Link | Class hierarchy | Pending |
| SpotBugs | Link | Control flow structure | Pending |
| SonarQube | Link | Control flow structure | Confirmed |
| SonarQube | Link | Variable declaration | Confirmed |
| SonarQube | Link | Variable declaration | Fixed |
| SonarQube | Link | Variable declaration | Fixed |
| SonarQube | Link | Compound expression | Fixed |
| Infer | Link | Class hierarchy | Pending |
| Infer | Link | Variable declaration | Pending |
| Infer | Link | Control flow structure | Pending |
| PMD | Link | Variable declaration | Fixed |
| PMD | Link | Control flow structure | Fixed |
| PMD | Link | Compound expression | Fixed |
| PMD | Link | Control flow structure | Pending |
| PMD | Link | Compound expression | Pending |
| SpotBugs | Link | Variable declaration | Pending |
| PMD | Link | Class hierarchy | Pending |
| PMD | Link | Class hierarchy | Pending |
| PMD | Link | Compound expression | Pending |
| PMD | Link | Variable declaration | Confirmed |
| PMD | Link | Compound expression | Pending |
| PMD | Link | Control flow structure | Pending |
| SpotBugs | Link | Compound expression | Pending |
| PMD | Link | Variable declaration | Pending |
| PMD | Link | Variable declaration | Pending |
| SpotBugs | Link | Class hierarchy | Pending |
| PMD | Link | Compound expression | Fixed |
| PMD | Link | Class hierarchy | Pending |
| SpotBugs | Link | Class hierarchy | Fixed |
| PMD | Link | Control flow structure | Pending |
| PMD | Link | Control flow structure | Fixed |
| SpotBugs | Link | Variable declaration | Fixed |
Paper
Please refer to the author's homepage to check the camera-ready paper: [Link]
Open Source
Open-source [Link]
Timeout Selection
Our evaluation is based on the effectiveness of finding bugs in each static analyzer. Intuitively, a test generation approach can discover more bugs given a longer execution time. However, due to limited resources, we would like to investigate the appropriate time limit for running the evaluated analyzers. To determine the appropriate time limit timeL in Statfier, we perform an experiment investigating the effect of changing timeL on the number of bugs found. When running Statfier on each analyzer, we gradually increase the execution time limit on our evaluated analyzers.
The above figure shows the relationship between the number of discovered bugs and execution time. As our experiment shows that six hours is sufficient for finding bugs in all evaluated analyzers (i.e., the number of bugs does not increase after that), we select six hours as the time limit for running each analyzer in our main experiments.