When I started working for IBM’s Emergency Response Team I was a little intimidated about walking into a client’s environment and quickly providing incident response leadership. Luckily I was trained by Chris Pogue and Harlan Carvey to consider three things when I got on-site:
- What are you trying to answer?
- What data do you need to answer it?
- How do you get that data?
Obviously the first question is the hardest to answer (although, perhaps, not the most challenging). I often find myself having Car Talk-style conversations where you talk about the situation for an hour or two and then somebody says something to the effect of: “Oh, yeah. And we have this service running over here.”
It always amazes me how similar many of the security issues are across companies with decent security programs and teams. The security teams work hard to identify situations before they happen but incidents still occur and the teams struggle with pulling all of the information together. Disciplined incident response comes with experience. Not just experience within the security team, but experience throughout the organization. Most companies have information technology (IT) divisions with silo-ed experience (I am not saying it is bad, just natural). The web application developers and administrators know the web applications and web logs. Database administrators know the database logs, how verbose they are, how to retrieve them, and what can be turned on and off. Network administrators know what is logging where, what has been disabled, what can be turned up. Etc, Etc. Of course there is some overlap, but this example demonstrates that a lot of internal interaction may be required to initiate an incident response.
This silo-ing can complicate things but it is often manageable. The situations that prove to be more challenging are those that involve external services and equipment. Security teams are not usually consulted when service level agreements with external entities are formulated. Any third-party organization worth their salt will be prepared to provide information and action during critical times. But experience is the only way to determine the proper channels and methods of request to initiate the actions necessary to facilitate an efficient incident response. Acquiring data, removing services, increasing logging, and physically accessing a facility are just a few of the things that could increase the gap between incident identification and response.
Another thing that comes into play during an active incident response is the review of unique data types. I have assisted in several instances where an incident involved a web application running on an Apache server. The Apache web logs did not include any POST information which may have provided artifacts relating to the incident. As these web logs were the primary method for detecting anomalies the security team quickly realized that with the default configuration they may have been missing a few things. Digging a little deeper by including web application administrators and developers additional logs were identified. In a recent case the development team used Apache log4j to produce excellent activity information for debugging. As it did not introduce a performance hit to the server the developers just left the debugging code alone. The end result was a detailed log of the web server activity. The only real hick-up was that each entry was undocumented and multi-lined making it difficult to review by common automated processes. But, in the end, very helpful.
So, again, we come back around to the same incident response-related conclusion: Preparation is KEY. Understanding how things interact and who is involved is critical to reducing the time involved during each step of an incident response. But where to start with preparation? The results of an actual incident or a penetration test provide excellent examples and supporting data. Incident response scenarios are also very helpful but can be difficult to design and get everybody involved. To be honest, I have been a big advocate of “incident response scenarios” and I have told many people to develop them. But, looking back, I realize how hard it is for an organization to do this. Heck, it was hard for me when I was specifically thinking about and devoting time to developing a good incident response scenario. Therefore I am going to change my future recommendations from developing incident response scenarios to performing system walk-throughs.
I made this decision when I found myself leaning back in a chair, looking at a white-boarded diagram of a system (by this I mean a group of resources combined for a specific purpose), and trying to determine what we needed to understand what might have happened during an incident. It was excellent because I also found myself and the security team asking questions to which we did not have immediate answers. I also found myself thinking about how I would have accomplished a specific set of actions relating to the incident. Coming up with these possibilities lead me to asking additional questions and identifying other resources-of-concern and data that could be used to identify useful network and system artifacts.
Therefore, my recommendation is that security teams periodically perform a system walk-through for various IT implementations in their organization. See how systems walk-throughs work and let us know your experiences. Pick one thing at a time when you start and see where it takes you. Focus on the things that will provide you information about a specific activity. Brain-storm situations and talk about the system and network-based artifacts that would be helpful. Folllow up on these ideas and pull in personnel to provide additional information when necessary. This will increase their experience and begin the collaboration process that may prove vitally important in future incidents. I also believe that these interactions will help identify risks, weaknesses, and vulnerabilities that can be easily addressed and increase the overall security of the organization.
Go forth and do good things,
Don C. Weber