My Security Planet » Matasano Chargen

Matasano Chargen

Ruby For Pentesters - WIN32OLE

Posted: September 26th, 2009, 6:43pm CDT by Chris

Tags: [edit]

This week in our Ruby For Pentesters series I wanted to cover a Ruby library we have used a lot over the past year or so. Using Ruby on Windows is not one of the most exciting things I can think of. But there are times when you have no choice, working with ActiveX controls is one of those times.

Working with COM objects can be tricky, its a complex and sometimes confusing technology. Lucky for us, Ruby provides us with a library called WIN32OLE by default. WIN32OLE can be used for parsing and controlling COM/OLE components from Ruby. For example, we can use the WIN32OLE library to open up and play with Internet Explorer.

    require ‘win32ole’

    ie = WIN32OLE.new(‘InternetExplorer.Application’)
    ie.visible = true
    ie.navigate(‘http://www.runplaybook.com’)

We passed in the ProgID for Internet Explorer and sent it to a URL of our choosing. Pretty standard stuff for a COM interface. But it’s pretty neat how seamlessly it works form Ruby.

Do you subscribe to milw0rms RSS feed? Its full of ActiveX vulnerabilities. But let me fill you in on a little secret, most of them are nothing more then feeding some random ActiveX controls eatAString() method a bunch of A’s. But how are these obscure (it’s called sarcasm people) and seemingly exploitable bugs found?

Well I’m here to show you! And I promise by the end you will be finding bugs in some random ActiveX control on your box. When an ActiveX control is marked ‘Safe For Scripting’ it usually exposes a bunch of methods and properties that the browser can call or set from a scripting language like Javascript or VBScript. This is what makes ActiveX bugs fun: ITS NATIVE CODE. But how can we find these interfaces and start poking at them with Ruby? WIN32OLE of course.

Note: We won’t go too deep into the details of COM/OLE here, check out [www.cert.org] or [uninformed.org] if your interested in a more detailed write up.

WIN32OLE can do a lot more then just control InternetExplorer. We can use it to list all of the properties and methods any ActiveX control exposes. Lets drill down and focus on an individual ActiveX control that we can start fuzzing. Microsoft XP ships with an ActiveX control named ‘htmlfile’. You can read more about it here [cometdaily.com] and here [msdn.microsoft.com]. This is a good example control because it contains a lot of methods we can play with. Heres some small example ruby code that demonstrates how to get a list of methods the control exposes:

   require ‘win32ole’

   a = WIN32OLE.new(‘htmlfile’)
   methods = a.ole_methods.select { |m| m.visible? }
   methods.each do |meth|
       puts “#{meth.name}(” +
           meth.params.map {|p| “#{p.ole_type} #{p.name}” }.join(‘, ‘) + “)”
   end

You should have seen a bunch of junk scroll up your terminal, junk like:

   cloneNode(BOOL fDeep)
   removeNode(BOOL fDeep)
   swapNode(IHTMLDOMNode otherNode)
   replaceNode(IHTMLDOMNode replacement)
   appendChild(IHTMLDOMNode newChild)

What you saw was WIN32OLE opening the ‘htmlfile’ control by using its ProgID and dumping the methods exposed by the control along with type information. Great, now we know what methods we can call into and what type of arguments those methods are expecting. With this information we can start generating test cases and looking for bugs. But we’re getting ahead of ourselves here, first we need to understand how to instantiate the control in a real-world way. We can use some simple HTML and Javascript for that:

   <html>
    <script lang=’JavaScript’>
       var axobj = new ActiveXObject(“htmlfile”);
    </script>
   </html>

Note: Yes, its possible to fuzz directly into the control via WIN32OLE but were interested in vulnerabilities we can reach via Javascript within Internet Explorer. For this reason we stick with the ‘fake webserver’ technique.

Now if we wanted to call the cloneNode() method within htmlfile our Javascript looks like this:

   <html>
    <script lang=’JavaScript’>
       var axobj = new ActiveXObject(“htmlfile”);
       axobj.cloneNode(1);
    </script>
   </html>

So now we need to use Ruby to automate all of this and find some bugs. Enter AxRub.

AxRub is a tool I threw together and discussed this July at Blackhat 2009 during our Ruby For Pentesters talk. AxRub was inspired by HD Moore’s AXMan, which is an impressive tool, but difficult to use on a targeted penetration test. AxRub makes the process of fuzzing ActiveX controls much more targeted, and fast! It’s very early stage code and needs a lot of improvement, your ideas are welcome. You can grab AxRub here http://github.com/struct/AxRub

Here is an overview of how it works:

   1. Use WIN32OLE to get a list of methods/properties our target ActiveX control exposes
   2. Setup a small fake web server
   3. Listens for connections from IE
   4. Generate test cases and serves them up via HTML

Demo:

   C:/> ruby axrub.rb htmlfile

Now connect to http://localhost:8080 with Internet Explorer and wait for those quality 0days to roll in.
Indie Software Security: A ~12 Step Program

Posted: September 24th, 2009, 3:02pm CDT by Thomas Ptacek

Tags: [edit]
Every autumn, John “Wolf” Rentzsch holds an indie software development conference for Apple developers in Chicago called C4. It’s really excellent. I’d recommend you attend, but it’s become one of those things that sells out the day he announces the tickets. We don’t get to go this year.

But last year, we did get to go. Because we’re local, Rentzsch asked us to get up on stage and give a talk. So we roped in Nate McFeters, another local, and put together a security talk for indie Mac developers with no budget for security. What does a security talk for Mac developers look like? As it turns out, it’s very much like the talk we think every indie developer, Mac or not, should see, and it’s very much unlike the talk the rest of the security industry is giving. And, without further ado:

Here’s our talk. Here’s the slides.

(But see the bottom of this post for some caveats about this talk.)
Indie Software Security: A ~12 Step Program
You’re launching the first version of your web application. You’re pre-revenue. Your every waking moment is consumed with the backlog of product enhancements you believe will help your app break through. And you’re about to land on the top of Reddit because of a security flaw.

We’re software security people. Our industry is built around breaking software. People like us are the ones putting people like you on the top of Reddit. People like you are the weak, and people like us are the tyranny of evil men.

You’re listening to us because you’ve finally given up trying to control the security of your application. Everything else you’ve tried has failed. The advice the “security industry” has given you has had negligible business value, because you’re not a Fortune-500, and you aren’t shipping shrink-wrap to 1,000 enterprises. And the failure of that advice is partly our fault. This is our response.

Read the advice we’re giving here. See how you’re doing. Honestly, are you on top of these issues? Remember, there is no disgrace in facing up to the fact that you have a problem.
Step 0: Are You Our Audience?
Do you make lots of money? Do you have lots of money? Are you taking credit card numbers? Do you have a formal SDLC? Do you actually employ security researchers? Does anyone in your company have “security” in their title? Answer “yes” to any of these?

You’re not our audience. You have different problems. Some things we say you should do in this post, you should not do. Other things we say not to do, you can go ahead and do.

With that said, critique away.
Step 1: Stop Caring So Much
Here are five things that will kill your startup before software security does:
1. Slowness
2. Poor graphic design
3. XML
4. The RIAA
5. Product Marketing Managers
The graveyards in this town are littered with the corpses of startups that pinned their hopes on advanced security. Better engineers than you have tried and failed. Theo de Raadt coordinated the first large-scale security codebase audit. His reward:

Three years without aOnly two remote holes in the default install!

This is not a killer marketing message.

Or, try Daniel Bernstein. qmail shipped for ten years without an exploitable remote flaw. It has the best security track record of any piece of mainstream software, ever. But Bernstein got integer signedness wrong in one place —- resulting a bug that you can’t even exploit on any modern system —- and now every security blurb about qmail is up for debate.

We don’t want you to be the guy in the PG-13 movie, the one everyone’s really hoping makes it happen.

37signals is not bad about software security. They’re incredibly successful. They make millions of dollars on an online contact manager. 37signals has a genius for marketing. They’re so money! It’s like Jedi Mind Shit! They’re not bad about security; they’re just not awesome about it.

We want you to be like the guy in the rated R movie. You know, the one you’re not sure whether you like him yet. Okay? You’re a bad man. You’re a bad man.
Step 2: First, Get Outreach Right
Do this first. Don’t waste time with anything else.

There are a lot of ways to be good at security without being awesome at it. Most of them don’t matter. This one does: you need to be smart about how you interact with people finding and reporting vulnerabilities in your products

Take, for instance, 37signals. There’s been a pretty negative response to their handling of the disclosure of a cross-site scripting vulnerability in their software. Why did that happen? Because if your company doesn’t act as if it knows how to handle security reports, the world will assume the worst of you, no matter how hard you’re trying.

This is literally the simplest security challenge your shop shouldn’t be screwing up:
- Get your developers in a room, draw straws, and the short straw is now “security@mystartup.com”. She’s the “security contact”. You now have one.
- Create a “security” page on your main site. It says, in a nutshell, “we’ve thought about security.” It does not talk about AES key sizes or RSA-OAEP.
- Create a “security reports” page. Link to it from your “security” page. It says, in a nutshell, “here’s how to send us security reports, and we’re not going to be jerks about it.” The one 37signals set up is, we think, an excellent example of the genre.
- Your “security reports” page needs to link to a PGP key. This is code for “we’ve given a moment’s thought to the idea that we might have a security flaw in our app reported to us”.
- Fixes for security flaws are not features or enhancements. Don’t call them that. In fact, don’t even call them bugs. Give them “security IDs”. This costs you nothing, and is another subtle signal you can send that you’ve done this before.
- For the love of god don’t argue about severities. You won’t win. Security researchers have more buzzwords to throw than you do. Even if you’re right, 50-75% of readers will assume that (a) you’re not right and (b) you’re being petulant.
- Thank the people who submit security flaws to you, even if you don’t feel thankful.
If there’s one thing we want you to understand about vulnerability reporting, it’s this: however hard your startup has had to struggle to get press and attention, no practicing security researcher has had to work 1/100th as hard to get in the mainstream press. For a myriad of reasons, some good, many bad, everything security researchers do is newsworthy. They’ve drawn KK to your A7 suited. Don’t overplay it.
Step 3: Everything You Need To Know About Vulnerabilities In Two Bullets
There’s a sprawling literature about different kinds of security flaws. You can safely ignore it. If you try to stay up to speed, you’ll still lose, but you’ll miss the simple stuff. You only need to know two things, and here they are:

Bullet 1: Counting is scary. Many billions of dollars have been spilled dealing with one basic problem, which is that it’s hard to keep track of memory. If you understand this, you understand stack overflows, heap overflows, integer overflows, integer underflows, uninitialized variables, offset null-pointer attacks, and even that crazy pulseaudio Linux kernel flaw that depends on a compiler optimization.

Bullet 2: Content is scary. Many billions of dollars have been spilled dealing with one basic problem, which is that it’s hard to keep metacharacters out of user-controlled content that moves between different domains. If you understand this, you understand cross-site scripting, SQL injection, shell injection, xpath injection, and Nate McFeters GIFAR attack.

And so:
- Initialize your variables to 0 or the empty string.
- Abort your program when malloc fails.
- Count, using unsigned integers, and don’t let them wrap.
- Whitelist content to alphanumeric.
- Swap punctuation for HTML entities.
- Go live your life.
But, but, but! What about that Black Hat talk that Mark Dowd gave about the plug-in interfaces in all those browsers? What about that crazy Flash exploit he wrote? Those were cool. They made waves in the security industry. But they weren’t about you. His audience is security researchers, not indie startups.
Step 4: Seven Deadly Features Of Indie Software
Here are 7 features that you simply shouldn’t implement yourself.
1. Encryption, unless it’s GPG for data at rest, or SSL for data in motion.
2. Password storage, unless it’s bcrypt.
3. Writing anything directly into the DOM of a browser via a plugin or any third-party software.
4. Installers.
5. Things that listen on network ports when your code is running, or (just as likely) all the time. Even —- no, wait, especially —- if it’s a web server.
6. Content-controlled code. For instance, if you’re designing a web templating system for a PHP app, the templating language cannot be PHP.
7. File download or, worse, upload.
People hear this list and they think we mean “be really careful when you implement these features”. We do not mean “be really careful”. We mean “don’t do it”. Change your app design so it doesn’t need the feature. If you can’t get dodge the feature, find its best-known, most-beloved implementation, and use that instead.
Step 5: Deploy Rubber Chicken Security
Here are some security features that don’t really do anything for your security, but that you should consider doing anyways:
1. Big long random URLs. Nothing says “security” like a big long random URL.
2. SSL. Use it wherever you can, and then you get to put the little graf on your “security” page about how you use “the same kind of security that banks use”.
3. Little lock icons.
4. Encoded inputs and outputs. Don’t just stop at Base64; jumble the Base64 character set up, so when people decode it, it looks like it’s been encrypted.
5. Encryption. Wait, didn’t we just say not to use encryption? Yes. Don’t rely on encryption. Don’t care about the encryption. But do scramble things. And don’t just use AES; add or subtract a couple of rounds (a 1-line change in your AES code), so that a standard AES implementation won’t decrypt things properly.
6. Write things in a “safe” scripting language instead of C.
7. Buy “Hacker-Safe” certification, or get PCI certified, so you can put a little seal on your site.
None of these things really protect you. For every item on this list, there’s a domain-specific threat that’s far more important than what that feature is trying to address. Some of these features do nothing for you whatsoever.

But that doesn’t mean you shouldn’t take advantage of them. Some of them improve your marketing. Some of them raise the bar, very slightly, on finding flaws in your site, which may somewhat improve the quality of security researchers you deal with. Just don’t talk about this stuff in arguments with security people, and you’ll be fine.
Step 6: Fuzz
And now it’s time for Secrets of the Security Masters.

Every year, the security industry gets together at Caesar’s Palace in Vegas for Black Hat, the most important conference in software security. A huge chunk of the vulnerability research community submits talks. Most of these talks get significant press. And 50% of them follow a predictable pattern:
- Here is a new technology that is in the news or that secretly controls our lives.
- Here is the fuzzer I wrote for this important technology.
- Here are the horrible flaws I found when I ran that fuzzer.
(Hey, we’re guilty as charged here).

What’s a fuzzer? Very simple. A fuzzer is something that knows how to talk to something else, using a protocol or a file format or an SDK, and that systematically replaces parts of the input with progressively scarier inputs —- long strings, SQL metacharacters, long strings seperated by various forms of punctuation, long strings seperated by SQL metacharacters, the 4 numbers clustered around every bit position in 16, 32, and 64 bit numbers, and so on.

You’re a software developer. You can write this code. For instance, part of your application handles vCard contacts. Write the vCard contact fuzzer. Run it against your application. If you wrote your fuzzer the same way most consultants write theirs, it will take a day or so to run. Fix what it finds.

If you’re a web developer, you’re in luck. Somebody wrote a really good fuzzer you can just go buy for a couple hundred bucks. It’s called Burp Suite. You can about the “Proxy History”, “Repeater”, and “Intruder” tabs. Buy it, run your application through it, and figure out how to use those these Burp features. Cover every input in your application. That’s almost 90% of what every pro web-app pentester is going to do.

(You want the commercial version, not the free version.)
Step 7: Your Code Isn’t A Secret
This last point is easy, and you’re probably already on the same page as us.

A couple years back, Nate Lawson reverse-engineered the Bay Area FasTrak tolling system, which uses an RF protocol to tell tollbooths to debit your toll account. To pull this off, Nate got a FasTrak dongle and reversed the hardware, at one point going as far as decapping the chip to enable the JTAG debugging interface.

Nate Lawson is an extremely smart guy. But he has a bill rate. He’s part of this industry. Any company that wants to work with him can pay him, and he’ll do that same stuff to whatever app they sicc him on.

My point here, and I do have one, is that you are doomed. The state of the art in reverse engineering has advanced to the point where, on general-purpose computers, it’s mostly point-and-click.

Nothing in your software should depend on the security of your source code. Even more importantly, you can’t fetishize your source code. If you communicate to the world that your code is hard to reverse engineer, then it’s a “win” for a security researcher just to have reversed it, whether or not they find anything interesting from doing so.

If you’re shipping code written in anything other than C, you should understand that you have shipped your source code. Even if you ran it through an obfuscation product of some sort (another rubber chicken). An easy way to absolutely convince people who know security that you don’t know anything about security is to make an argument that involves how hard it would really be for an attacker to figure out how your software works.
Step 8: And that’s it.
We’re short some steps, and that’s because for an indie development shop, those steps don’t matter. So few people are getting this stuff right that it seems pointless to talk about the rest of it. And that’s sad, because what our program recommends costs almost nothing, doesn’t involve certification or third-party testing, and immediately improves outcomes in security incidents.

Stop freaking out about security. You’re worrying about the wrong things anyways. Get this simple stuff right, and then get on with your life.
Some quick caveats about the talk itself:
- We gave this talk a year ago.
- We went back and annotated the slides, which really don’t stand by themselves, and we know that too.
- YES I HAVE ARMS. It was the only clean button-up shirt I had that day.
- Again, this is a talk aimed at indie Mac developers. If you’re a large ISV, a bank, a Fortune 500 company, we know: you do not want, you cannot use.
Ruby for Pentesters: Stupid FFI Tricks

Posted: September 22nd, 2009, 2:25pm CDT by Eric Monti

Tags: [edit]
Lately, Ruby has left me wondering whether I should go back to C or (gasp) python for my day-to-day coding. But then I took a breath of fresh awesome that convinced me to stay put.

Here’s a fun trick I stumbled into while playing around with FFI and Metasm in ruby.

This probably falls under the “stupid ruby hacker tricks” category. I’m also pretty sure the FFI guys never had this in mind as an exposed feature in their library. But damn if it isn’t refreshing when sometimes things work exactly as you hope they will.

When I see something like this work, it also occurs to me that my reluctance to leave the fireside coziness of my IRB prompt is a sign of something. Probably something unhealthy.
Scenario:
You’re coding in assembly language (for good, evil, or neutral) and you’d like to quickly test your code as you work. You know what would be great? It would be great to have a way to dynamically assemble and shove your instructions into memory someplace, then call them on the fly.
Solution:
Write a little wrapper to load your bytecode into memory and jump to it. No fussing with ELF/PE/whatever headers, overwritten return addresses on the stack, heap bugs, etc. This isn’t necessarily exploit development, just shellcode development. The goal is just to make sure your bytecode works the way you expect it to when you land on it.

So… you could:
Abuse a function pointer in C:
Here’s how you could do this in C with minimum fuss. Use malloc(3) and memcpy(3) to load your code from a command-line argument and cast the resulting heap memory address to a function pointer. Then call the function pointer.
```
    #include <stdlib.h>
    #include <string.h>

    int main(int argc, char *argv[]) {
      size_t bloblen;
      char *blob;
      void (*funcptr)();

      if (argc > 1)  {
        bloblen = strlen(argv[1]); // hope you're nullsafe!
        blob = (char *) malloc(bloblen);
        if(blob != NULL) {
          memcpy(blob, argv[1], bloblen);
          funcptr = (void *) blob;
          funcptr();
          exit(0);
        }
      }
      exit(1);
    }
```
Or Abuse a FFI function pointer in ruby:
Here’s how you can (ab)use FFI from ruby to achieve the same effect. This reads the code from standard input instead of the command-line, by the way:
```
    begin ; require 'rubygems' ; rescue LoadError ; end
    require 'ffi'
 
    # Use FFI to stuff our bytecode somewhere on the heap.
    # here's the malloc(3)/memcpy(3) combo
    code = STDIN.read
    memp = FFI::MemoryPointer.from_string(code)
 
    # memp is now a pointer object FFI can use

    # Now we cast our function pointer.
    #
    # Yea so this is much nastier looking than:
    #
    #    void (*funcptr)();
    #    funcptr = (void *) blob;
    #
    # It'd be swell if FFI stopped changing this interface, but
    # hey, beggars can't be choosers. I'm not complaining...
    funcptr =
      ## use FFI::Function for ffi-0.5.0.
      if FFI.const.defined?("Function")
        FFI::Function.new(
          FFI.find_type(ret), args, memp, :convention => :default )

      ## use FFI::Invoker for ffi-0.4.0 - two flavors even!
      elsif FFI.const_defined?("Invoker")
        if RUBY_PLATFORM=='java' ## JRuby FFI
          FFI::Invoker.new(memp, args, FFI.find_type(ret), "")
        else ## and not Jruby...
          FFI::Invoker.new(memp, args, ret, FFI.find_type(ret), "", nil)
        end
      else
        raise "oh noes! this version of ffi is totally unfamiliar"
      end


    # Now we call our bytecode stub directly.
    # This is basically like saying "funcptr();" in the C version.
    funcptr.call()
```
FFI rocks in general. Ruby’s been lacking a good answer for python’s ctypes, and FFI is definitely the answer. But I’m stalking Wayne Meissner to make sure he keeps this feature around and maybe even gives it a standard interface.

As evidence that I’m mentally unhinged, here’s a version of that script with all kinds of superfluous and ridiculous additional features.
```
   Usage: asm_lab.rb [opts] < someassmebly.s
          -s, --sled=SIZE         Add a nop-sled
          -g, --debug             Add debug trap and spawn gdb in xterm.
          -f, --file=FILE         Read input from file instead of stdin
          -r, --raw               Input as raw bytecode
          -d, --drop_id=RID       Drop real privs to RID
          -D, --drop_eid=EID      Drop effective privs to EID
          -h, --help              Show this message.
```
Um, a not so superfluous feature there: metasm!

Metasm rocks! ‘Nufsaid.
Take Survey, Get Free Huge Poster, Courtesy Of Me

Posted: September 14th, 2009, 5:00pm CDT by Thomas Ptacek

Tags: [edit]
OK, that’s 100 (actually a little more). We’ll still give away 50 more, at random, to people who complete the survey this week.

So, we made these posters. We meant to hand them out at Black Hat, but they didn’t get finished in time, even though we rushed them by —- among other things —- doing it ourselves without our designer, who will certainly mock us for the next year about the color scheme and perhaps you have a recommendation for a good print designer for us? And anyhow, here it is:

Yes, we have a fetish for hex charts. We also have “man ascii” in like 15 forgotten terminal windows on all our desktops. And now we can just look up at the wall. And here are some things to know about the posters:
- They are big, by shwag poster standards. About as wide as a normal cubicle panel.
- They are printed on heavy stock, which we thought would be a win but turns out not to be as much of a win as we thought but might be great for you.
- The image here is not 100% accurate, because Preview.app botched the PDF->GIF transformation, so please don’t submit bug requests about it.
- I am fully to blame for the terrible color scheme.
And you’re wondering, how do I get one of these posters? And I am here to tell you that today, one good way to get a poster is if you’ve ever been responsible for managing more than one firewall, you could: fill out our survey and help us figure out what to do next with Playbook, our firewall sync product.

Otherwise, you can wait, and we’ll come up with something else that will get you a poster shortly. I think they’re pretty cool, even if they are kind of crazy looking.

First 100 real submissions get free posters. Next 50 posters after that go out by lottery. We’ll let you know when that happens.

PS: I may already owe you a poster, in which case, drop me an email or a DM, and we’ll send one.
The Joel Test: 12 Steps To Better IT Management

Posted: September 14th, 2009, 2:59pm CDT by Thomas Ptacek

Tags: [edit]
(With apologies to Joel Spolsky, from whom this post was ripped off)

Have you ever heard of COBIT? It’s a fairly esoteric system for measuring how good a network security team is. No, wait! Don’t follow that link! It will take you about $3,500,000 just to understand that stuff. So I’ve stolen my own, highly irresponsible, sloppy test to rate the quality of a network security team. The great part about it is that it takes about 3 minutes. With all the time you save, you can go to law school.

(I’m using network security as a synecdoche for all of enterprise IT, but I think I’d win an argument about whether the same issues apply to the team that manages the WebSphere deployments and all the WAR files and whatnot.)
The Joel Test
1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?
1. Do you use source control?
Programmers have had source control since the 1980s. With source control, every change you make to a file is tracked. Any line of code can be tracked diff-by-diff back to the origin of the file. Are your firewall rules under version control? If you don’t have source control, you’re going to stress out trying to get engineers to work together. They’ll have no easy way to know what their coworkers did. Mistakes can’t be rolled back easily. And source control backs up all your rules, in a single place, so you can reconsitute that PIX ASA that just threw a rod from a cold spare in minutes.
2. Can you make a build in one step?
By this I mean, how many steps does it take to get an access rule change deployed from the latest source snapshot? On good teams, there’s a simple process you can follow to get a firewall completely deployed from scratch, which checks out the company standard configuration, adds the right doodads to the configuration, tracks the device in inventory, etc.

A process that takes more than one step is prone to errors. And when you’re under pressure because Pepsi needs you to punch a hole for a database management app that Coke has forbidden you from allowing near their dat,a, you want to have a very fast cycle of making sure your rules work. If it takes 20 steps to deploy a rule, you’re going to crazy and you’re going to bring the network down.
3. Do you make daily builds?
When devs use source control, sometimes people check things in that break the build. The usual pattern is, things work fine on the developer’s machine, but she forgot to add a header file, so nobody else can build. “Breaking the build” can cost developers whole days of productivity, so good teams have rules to detect and punish people who do it.

Network security engineers have a “build”, but when they break it, they don’t just kill the rest of the team’s productivity. They kill the network. So good network security teams want to make sure people can work on projects that touch access rules without getting unreviewed changes into the configurations that will get pushed out during the next change window.
4. Do you have a bug database?
I don’t care what you say. If you’re managing access rules, even if you only have a few devices, without an organized database listing all the change requests that produced the rules, you’re going to have crappy firewall rulesets. Lots of engineers think they can hold the rules in their heads. Uh huh. What hosts on your network are allowed to talk FTP to the outside world, and why? Thought so. You absolutely have to track change requests formally.

Change tracking can be complicated or simple. A minimal tracking system needs to record the following facts for every request:
- Who requested the change
- Why they requested it
- Was the request approved
- Who was the request assigned to
- What devices had to change to accomodate the request
You can absolutely DIY this; lots of teams build their own tracking systems in-house, and they work great.
5. Do you fix bugs before writing new code?
When Joel Spolsky worked on the Excel team at Microsoft, he picked up a story about Word for Windows, which slipped constantly because the schedule allowed no time to fix bugs. The quality of the codebase decayed, to the point where developers were writing “if 2+2 !+ 4 return 4” or whatever. They referred to this as “infinite defects methodology”, converted to “zero defects methodology”, which meant they got to fix bugs, and eventually shipped. And when Joel Spolsky was in the alps fighting grizzly bears, he used his magical fire breath and saved the maidens fair.

But I digress.

Once every year or so, big companies commission small companies like ours to do the “annual external pen-test”, in which testers try to break in through the perimeter firewall. Even though I don’t do a lot of network pen-testing, I’ve done a couple. And on all of them, some stale old Win2k host gets left exposed or some branch network has 445/tcp open, because there are 20,000+ lines of firewall rules and rules only get added, never removed.

Just like with code, it is much more expensive to fix a bug early than late. But at least with software, you find out about bugs because your program crashes or a user sees the wrong header font size in the help file. With firewall rules, not so much. You mostly find out that you’re boned when you flunk some random audit.

Most of the same reasons developers need to fix bugs before writing new code apply to enterprise IT, too:
- It’s easier to fix a bug when it’s right there in your face than to remember it or track it down later
- It’s easier to predict to your customers when their changes are going to get completed when you know you aren’t going to lose 2 days fishing old SMB exceptions out of your rules after an MSRC announcement
- It’s less stressful and less costly to fix things up front than to blow a change window or push an emergency change because of an advisory or a client audit.
6. Do you have an up-to-date schedule?
Which brings us to schedules. Because the dirty secret is, the rest of the IT team at your company probably hates you, because your job mostly involves saying “not yet” to their requests to change things on your devices. But that doesn’t cut it. Too many business processes are impacted by your workflow; clients can’t get brought online until the PMP-certified project manager checks off your box in the process, and so you’re a cost center, and the VP/Operations starts to hear about how one of next year’s priorities is to “streamline firewall management and reduce TCO”.

It’s possible to keep a schedule; figure out how fast you can complete a change, and track the number of outstanding change requests you have. It can take significant time to research and execute a complicated firewall architecture change, as long as you can communicate up-front to project teams how long it will take, and then come in on schedule.
7. Do you have a spec?
Documenting all your configuration is like writing a software spec: everybody agrees it’s a good thing, but nobody does it.

It’s weird that this is the case, because nobody in the company has a stronger opinion about how technology should be deployed and managed than the network security engineering team, but probably all you have is apocryphal Visio diagrams on a network share somewhere, and you’re much more likely to just throw another configuration line onto a device than to write a document that explains what you’re doing.

Documentation doesn’t have to be painful. You can just set up Mediawiki and start by writing a short paragraph for every device you manage. Or you can write programs that take inventory and analyze your configs. However you do it, you should be able to have a rule that says “no changes without updating the documentation”.
8. Do programmers have quiet working conditions?
And also, pet unicorns?
9. Do you use the best tools money can buy?
“Top notch development teams don’t torture their programmers […] and programmers are easily bribed by giving them the coolest, latest stuff”.

Things it’s crazy network security engineers don’t get to have include:
- A network switch on their desk
- Two monitors
- A laptop with a big screen
- Unlimited disk space
- A place to get a new VMware image up with a couple of clicks
- A license for Burp Suite
I could go on and on here, but the only reason I’m writing this is that Joel Spolsky has this as a Joel Test item. Motherhood, meet apple pie, and also see “pet unicorns”.
10. Do you have testers?
“Skimping on testers is such an outrageous false economy that I’m simply blown away that more people don’t recognize it”, which is probably why most network security teams have testers. There had to be something network engineering teams do better, processwise, than developers.
11. Do new candidates write code during their interview?
It’s hard to write code in an interview. Interviewing teams are so infamous for skipping this step that there’s a whole interview question methodology, the “FizzBuzz test”, that says “at least show me you can print the numbers 1-100 with every 3rd number as ‘fizz’ and every 5th as ‘buzz’”, which is a 1-liner in Ruby, but that’s how desperately teams need to know that candidates even know where the parens and the braces go.

And so here’s another thing network security teams do better. Because, do security engineering teams have this problem? Probably very few candidates actually know how TCP flow control works, or whether they actually need path MTU discovery enabled, but I tend to doubt that a lot of teams are staffed with people who couldn’t punch an exception into an IOS ACL set for FTP or DNS if they needed to.
12. Do you do hallway usability testing?
And here I concede that there is one item on the Joel Test that does not directly apply to network security teams.
My Point, And I Do Have One
There’s a seed funding firm called Y Combinator that is all the rage with the kids these days; they’ll give you ~$20,000 for your 2-person startup in exchange for 5-6% of the company even if you have almost no working code and you’re just out of school. Some surprisingly good companies have come out of YC. One of them is Dropbox, a hugely popular and powerful file synchronization system. A few days ago, Dropbox’s application to YC was posted. One of the grafs in the application talked about Subversion, and ended with “hackers [(developers)] have access to these tools, but normal people don’t”.

And its true of network engineering teams too. In a lot of shops, Subversion is space alien technology (and yes, it’s true of some dev shops, too). And where problems are recognized, as with issue tracking, they’re “solved” by massive enterprise management systems with their own headcount dedicated just to keeping Remedy working properly, and everyone hates it.

And of course this is a self-serving post, because “solving the Joel Test for firewall admins” could be the thesis statement for our product. But then, I’ve recently changed up roles at Matasano, and am managing Playbook instead of consulting, and I want to start talking more about what we’re doing. And here’s a way to start the conversation.

Good to be talking to you all again, by the way.

← August 2009 (7 items) October 2009 (5 items) →

My Security Planet » Matasano Chargen » September 2009

Feeds

Matasano Chargen

Ruby For Pentesters - WIN32OLE

Posted: September 26th, 2009, 6:43pm CDT by Chris

Tags: [edit]

Indie Software Security: A ~12 Step Program

Posted: September 24th, 2009, 3:02pm CDT by Thomas Ptacek

Tags: [edit]

Ruby for Pentesters: Stupid FFI Tricks

Posted: September 22nd, 2009, 2:25pm CDT by Eric Monti

Tags: [edit]

Take Survey, Get Free Huge Poster, Courtesy Of Me

Posted: September 14th, 2009, 5:00pm CDT by Thomas Ptacek

Tags: [edit]

The Joel Test: 12 Steps To Better IT Management

Posted: September 14th, 2009, 2:59pm CDT by Thomas Ptacek

Tags: [edit]