Like Ken and Brian I’m pleased that results from this study are getting out there. I recently participated in the OWASP EU Summit, where Pravir and I conducted a session on the maturity model. The session had valuable industry participation.
One of the things I’ve harped on for years now is that experts need to pull their heads up from whatever they’ve gotten so good at doing in the weeds and avail themselves of some of the lessons learned by organizations that have been working on software security themselves. In 2006, I presented on “building one’s own software security group” and one of my first five slides uncovered a dirty little secret–true now as it was then: Fortune 500 security groups often possess more man-power and budget than security security consultancies. I see this maturity model study and the response/involvement of the people I’ve mentioned here as a giant step in the public recognition of the value security groups within large commercial entities have been providing.
To follow up on Gary’s InformIT article, I’ll be writing a series of posts on why these “Top 10 surprises” don’t come as a shock to us at Cigital and what it might mean for organizations. I’ll go in reverse order:
9. Not only are there are no magic software security metrics, bad metrics actually hurt.
First, a clarification. Gary’s article indicates that:
but even the most advanced programs don’t use any sort of balanced scorecard approach.
I certainly talk to organizations that rely on a scorecard, especially at the executive level, to report on software security activities. What’s interesting to me is the horror stories organizations tell about chasing the wrong metric. First, they lament how tracking “the wrong thing” motivated unexpected (and undesirable) behavior changes in the staff they were trying to effect. I’ve personally seen things like developers manipulating their code to break static analysis tools, signing a training sign-in sheet then leaving for the day, and vulnerability fixes that cause terrible residual risk (But pass a regression test in the form of a penetration testing exercise). Developers ‘hacking’ the system of control? We shouldn’t be surprised.
In my opinion, organizations charged the scorecard hill too early. They were frustrated with how “Fear-driven” security was but simply didn’t know what underlying measures would raise their visibility into key areas. Just as bad, they didn’t know how to take measurements precisely and inexpensively. I want to explicitly except organizations like GE from this analysis, BTW, as their rich history of scorecard management deserves its own treatment. For information on this, see GE’s presentation at Metricon 2.0.
Though there is often some pretty easy low-hanging fruit, leading organizations have begun to realize that deploying sensors to answer questions about software security spending is trickier than expected. Success involves looking at things on different levels than their budgetary line-items have defined. For instance, people are paying for penetration testing and they may have allotted money for a static analysis tool or code review services (or both). However, to determine and increase the efficiency of their assessment practices, organizations are finding more success with normalizing vulnerability findings reports, tying them back to what detected them, and generating “cost per findings” from there, rather than micro-optimizing each assessment technique itself. In my experience, this causes organizations to shift focus between assessment activities in ways that surprise them, but provide a profoundly cheaper overall assessment cost. As a result, organizations are surprised that a slight resurgence in penetration testing allows them to uncover authentication problems in their web applications more cheaply than the recently more favored static analysis tools are allowing them to. Likewise, simple customization of static analysis tools has allowed them to delete whole swaths of test cases that distracted or bloated their penetration testing efforts. More on this phenomenon can be found in my recent article.
I believe a scorecard will emerge. I believe metrics like “cost per finding” will begin to make sense and provide valuable decision support when organizations further mature the consistent application of their software security practices and realign the things they’re measuring. But, right now, great software security managers are conducting more “under the hood” diagnostics than “roll-up” reporting to increase effectiveness of their groups, reduce their cost, and weather this economic storm. No surprise there ;-)