In reply to Dinis's blog post: The Need for Standards to evaluate Static Analysis Tools
1. You unfortunately list few types of SAST. Many of tools don't implement taint analysis -- if you go in the Ada/C/C++ world, you won't see much of taint based analysis, but other technologies such as symbolic execution (Grammatech), abstract interpretation (ASTREE, PolySpace, etc.), and more. A list of SAST can be found on the NIST SAMATE website: List of Source Code Security Analyzers
2. As said on twitter, concerning the WASSEC, I don't believe it's important to have public evaluation of commercial/open-source tools. Also, WASSEC lists some vulnerabilities that the tool should look for, we don't provide test cases so it's not nearly possible to claim that a tool effectively test for a given problem, e.g. difference between two tools:
- Only test XSS with few payloads and does regexp matching of the rendered html
- A smarter engine that automagically crafts attacks and look at the resulting html with a JS engine (or so, that leads to fewer FP).
Depending on who you are and what you want, you might very well say that those two tools have the same support for XSS...
Moreover, tools are changing so quickly that an evaluation would only be accurate at the time you make it.
3. NIST SATE is literally an exposition. NIST choose test cases (real open-source program that covers different type of functionalities and technologies) and ask tool makers to run their SAST on those programs. The goal isn't to compare the tool to claim that one is better than the other for a type of techno, but it's too see how tools (in general) performs, to see how many types of weaknesses the tools find and also what is the overlap of tool findings (which resulted in a very little amount of findings).
More generally, as Andrew said, a SAST isn't only an analysis engine that finds weaknesses in a program; it's a suite of functionalities:
- support technologies
- allows users to develop custom checks (or custom rules)
- displays the weaknesses to the user (allow to rank/prune and explain problem) and reporting capabilities
Ultimately, every one of those elements are important and need to be tested, but again, the importance of those depend on who you are and how you want to use the SAST (from simple compliance type of scan to exhaustive security testing).
Just to tell you, NIST SAMATE (organizers of SATE) have been thinking a lot of those problem and there is no easy solution for evaluating SAST... But the last SATE report explains some of the problems we (I was part of the SAMATE team at the time) faced: SATE 2008 - NIST Special Publication 500-279