Starting this past winter, we tried an extended BSIMM-related experiment in self-reporting as a means of gathering software security activity data. We did this by directly contacting individuals and organizations to entice them to complete a survey. We called that effort BSIMM Begin. BSIMM Begin is related to the actual BSIMM, but it is not the same thing.
There were (at least) two basic approaches to gathering the software security activity data we were looking for. One was to ask what everyone was doing and then deal with the sorting and scoring issues presented by an avalanche of ad hoc data. Alternatively, we could ask about a few of the activities we had already seen in our controlled BSIMM surveys (http://bsimm2.com). We did the latter. On one hand, had we asked general questions about if and how software security was getting done, we might have amassed more data (assuming there were specific software security activities taking place). On the other hand, I believe we would have acquired a lot of data that had little to do with software security and much more to do with security features in software and with IT security.
Because we wanted some insight into a substantial number of data points, we asked a substantial number of questions. Because we didn’t want to ask overly-simple (and transparent) yes/no question about each data point and we wanted to have some confidence (intellectual, not statistical) in the results, we asked two or three questions about each data point. In the end, we asked a bunch of True/False questions, but did not include a Not Applicable option. That was a methodological mistake on our part and several of the respondents let us know that. Also, because we didn’t want the “right” answer to be “True” in every case, we “negated” some of the questions with a strategically placed “not” that really confused some respondents.
Overall, this resulted in a daunting survey that, anecdotally, took a full hour to complete, assuming the person taking the survey actually knew all the answers for his or her organization. If the person didn’t know all the answers, they had to stop and go discover some. Unfortunately, SurveyMonkey doesn’t really lend itself to stopping and starting surveys, so some people simply gave up. A couple of respondents let us know this somewhat colorfully.
We asked 110 questions to probe different aspects of 40 software security activities from the BSIMM—all of the Level 1 activities across 12 BSIMM practices and the few most common activities from Level 2. We received 93 responses, 48 of which were complete enough to allow useful analysis. As an aside, three of the 48 (and 22 of all) respondents declined to provide their email address, preventing us from sending their survey results.
Here are some general statistics on the data pool from the 48 respondents:
|
Avg. Count of BSIMM Activities Observed (out of 40)
|
Median Year Software Security Initiative Started
|
Average Number of Developers
|
Average Number of Testers
|
Average Number of Apps
|
|
15.15
|
2005
|
1342.19
|
251.79
|
1148.21
|
These are some fairly large numbers. Even though we went well out of our way to get smaller firms to respond to the survey, we had very little success in that area. One possible knee-jerk reaction is that there simply isn’t any organizational software security initiative in most of the smaller development organizations, but I don’t believe there’s enough data to make this a defensible conclusion.
The following chart shows the average software security activity level determined via the 48 responses. Within this small data set Penetration Testing and Software Environment saw the most activity, closely followed by Code Review, Standards and Requirements, Training, and Compliance and Policy. Security Testing and Configuration Management and Vulnerability Management were noticeably less populated.
In defense of this survey, I’ll note that a handful of participating full BSIMM firms graciously took the time to complete the questions as a scientific control. There was indeed some correlation between their self-reported data and the data we had meticulously gathered through in-person interviews. Informally, however, the results did seem to indicate that self-reported data result in consistently higher scores than results gathered in-person. That is, and as I’m sure is applicable in most walks of life, our own view of whether we are getting something done won’t always match that of an unbiased outsider.
The upshot of all this is that we will step up our efforts to do more in-person interviews in a full BSIMM capacity. The consistency in the data gathering and scoring will make the results more useful to everyone as a benchmark and a yardstick. If you’d like an understanding of where your software security initiative stacks up compared to others by participating the the BSIMM, we’d like to hear from you.
As a last point, we noted through personal knowledge or bounce messages that approximately 20% (10 or so) of the 48 respondents in the usable data set no longer worked at the same place they had just a few months earlier.
As always, feedback and suggestions are welcome.
So I began designing Remediation, a card game that pits web application companies focused on generating revenue against the threat of malicious users focused on negatively impacting that revenue for their own nefarious motives. This is the very scenario that we encounter at Cigital on a regular basis and a theme that can easily transcend the commercial world and find relevance within the Federal space… an area of particular interest to me due to my work with the