PHP analyzer correctness benchmark

· Repository

A low-cost, broad comparison of tool behavior. Not every benchmark expectation is desirable for every tool.

All results

OK: matches baseline. FP: unexpected diagnostic. FN: missing diagnostic. DIV: documented scope difference. ERR/BAD: measurement/case error. skip: not applicable. These are not universal quality scores.

Summary
ToolOK / judgedFPFNFP+FNExcluded

OK / judged = OK / (OK + FP + FN + FP+FN). Scope differences, errors and skips are excluded.

Method and discussion

Expectations invite discussion and corrections, not a prescribed feature roadmap or maintainer endorsement. External reproducers, corrections and contributions are welcome.

Cases are small semantic probes, not exhaustive upstream tests. Variants share a criterion ID. FP and FN mean disagreement with the stated baseline, including disputed warning policies, not necessarily implementation defects.

Passing CI confirms the measurement works, not that every expectation is met. Runtime checks cover the tested values, not all possible inputs.

Version changes require the same corpus fingerprint and PHP runtime. Failed or excluded measurements count as neither fixes nor regressions. The main build uses a full commit hash. Speed is not measured.

This selected corpus is not an overall ranking or a real-project defect rate. No automatic upstream issues, pull requests or maintainer contact.

Source coverage

Each lead maps to a reduced probe, an existing case or a reason for deferral. Coverage does not mean every upstream assertion was reproduced.

SourceCandidateOutcomeEvidenceOrigin