nd
Subscribe to the RSS feed

Keyword - Grabber

Entries feed - Comments feed

Tuesday, January 22 2008

PHP Source Code Analyzer

Months ago, I was talking about and doing some small tests with the php source code security analyzer that I was able to find on the web.

I was able to quickly test the new Fortify SCA 5.0 which is handling PHP application now. I can tell you that I am really exciting about this tool. First of all, it beats from far all the tools I've tested previously (for PHP), which is fair since it's a commercial tool.

But what I'm really excited about now is that I will be able to make more tests on my test suites, compare with my security metrics & basic security analyzer, looking at the behavior of SCA tools when the source code is obfuscated, and so on. You're on the good track Fortify, now, open an API and I will be able to make an hybrid tool...

Since I also have some plan of testing real PHP applications with both testing approaches (static/dynamic), I'd like to see the difference of application coverage, vulnerability finding and false-positive rates (yeah, the last one is obvious, but still interesting).

I'm also glad to see that vendors are taking PHP as a serious language and not only for script kiddies.

Friday, December 7 2007

Attack Surface oriented Crawler: Focusing on what you actually want

Previously, on the python version of Grabber, I used a BFS crawler. Good to scan all the code (as long as the parsers are not that dumb). The problem with these crawlers is that it's totally inefficient: all the problems are not everywhere.

Starting with this assumption, I tried to start rating what is actually important and what are the evidence that a page may be important for a security testing point of view. So, the architecture of the crawler is simply based on a priority queue and the priority is for now based on obvious reasoning which may be wrong: The script I prefer testing, is the one that is in POST, where the action is in HTTPS (and so on for the rest...) which gives something like that:

  priority <- 30
  If Form Then
    priority <- 10
    If Method = Post Then
      priority <- 5
  else if Anchor Then
    If Get Variables Then // To Understand: index.php?foo=plop, compared to index.php
      priority <- 20
  If HTTPS Communication for {Method action or Anchor URL} Then
      priority /= 2

This is a fairly incomplete work and kinda dumb, but at least it's unbiased for a set of URL.

Wednesday, November 21 2007

The new grabber

Grabber was a nice project. The main goal for me was to learn stuff around web application security/scanners; I didn't really know much before I started this project. But now that I've been playing with web apps scanners for more than 10months, I need to create a new one and go deeper in heuristics, browser integration and AI.

Grabber was in fact more a spider+fuzzer than something else... Not a good web apps scanner at all. Thinking of the analysis engine... It's something kinda stupid, no JavaScript execution, just simple heuristics for parsing and levenstein distances ;)

Anyway, I decided to start over this project. It's not gonna be a bunch of python scripts anymore, I am gonna use Qt/C++ extensively. The idea if this project is to be pen-testers oriented and open, I want to create a kind of wrapper around WebKit (especially using QtWebKit), a spider as core utilities and after, using plugins. The plugins should be either in C++ or JavaScript (QtScript actually). So far, we are 3 guys thinking of this project: we didn't start yet but we are open to every contribution; the project will of course be free and GPL'd.

I just post this in order to get some comments or suggestions about what a web apps scanner should do... Feel free to comment/mail...

Monday, November 12 2007

Interoperability and web application scanners

Talking about web application security scanners , we all have the same problem: False Positive. It's a fact that cannot exactly be solve by the testing methodology itself (since it relies on pattern detection). So, the idea I started talking about on #webappsec today is a common format for exchanging information between tools.

Ideally, this would work like this:

  1. Tool A is scanning a website.
  2. It exports some information a given format: out-tool-a.xml
  3. Tool B is able to understand out-tool-a.xml and take this as an input
  4. Tool B would then be able to verify the results/false-positive of Tool A by scanning with the information in the out-tool-a.xml

I really think that would be helpful somehow, at least for open-source tools. I'm gonna try to implement this for the next release of Grabber.

Monday, July 23 2007

Python script utility called wwwCall and Grabber news

wwwCall: HTTP(S) utilities

wwwCall is a very small module for Python (tested under python 2.5 but should be okay for python >= 2.3) which handle the HTTP(S) connection with some special features like proxy, cookies, authentification (basic, digest). This morning, I was working on Grabber and I just realized how ugly the code was, mostly because of how I handled the web connections, so I decided to create a simple module to do the job easily. The idea is to have a single object handling some basic function of the python urllib2.

If you have ever use Python for doing web calls, you'll see that the utilization is damn simple and I think, pretty cool... Example:

# create the object
http = wwwCall('http://rgaucher.info')
# add the features you want (cookies,auth)
http.setCookieFile('./the_path/file.cookie')
# reaching a logging URL and saving the cookie
http.post("http://rgaucher.info/login.php",{'username' : 'foo', 'password' : 'bar'})
# register the username/password for the basic authentification
http.setAuthBasic("romain","mypassword")
# print the content of the protected page
print http.get("http://rgaucher.info/401protected").read()

Download: wwwCall.zip

The next Grabber

So, I've been working on Grabber for a couple of months without a release now; it's mainly because I don't have that much time to work on it, but also because I made lots of modification. Today I added a couple of features:

  • Understanding some mod_rewrite rules for the spider
  • URL exclusion
  • Basic/Digest Authentification

This comes in addition on the previous features I added, mainly:

  • Multi Site
  • Multi threads
  • Cookie analyzer
  • XSS Locator in addition of the XSS Fuzzer which is definitely faster
  • Spider module, only to crawl the site and export it in XML
  • Login ability, keeping session state

I cannot give a d-day for the release of the 0.2 version because I really want to have a more stable product and will feed some test suites I made at work the tool, to be sure it's reasonable (I will not give comparison results with commercial products :P). I also want to have a better spider...

Monday, June 25 2007

How not to waste 6hours?

Make sure that your test case is correct!!!!!

Damn I'm stupid, I was working on Grabber on the session state management, and of course, I did a small test case with a couple of pages to be sure the spider can reach every pages. But, my test case was just stupid and calling twice my index make my session still alive, but the variables were set to an order just crazy and have the same effect as destroying the session.

Anyway, now it works! At least in the next Grabber release:

  • Multi site support
  • Multi-threading
  • Better Session state management, you can now add the login information in the configuration file
  • A new XSS detector based on few vectors and some variations on this. The XSS disclosure based on RSnake's Cheat Sheet is still here, but I needed a new one faster...
  • A module which makes Grabber be able to be used as a simple spider and will save the information in a XML file

I don't know yet when I'm gonna release the version, I need to make sure it works correctly and is stable, I also need to create something to generate nice report (maybe simple XSLT sheets developer/user side) and I want to work more on the hybrid mechanism using different tools (fortify,pixy,php-sat,swaat...)

Thursday, May 24 2007

PHP Source Code Security Scanners basic test

For quite a long time now, I've been playing with lots of different black-box tools: commercial or not, mine or not. Months ago, I developed Crystal, a plugin for Grabber which does the link between the black-box engine in Grabber and a PHP Source Code Security Analyzer: PHP-SAT . At the time, it was the only advanced PHP SCSA I could find on the web, so I used it without really testings I admit.
That's for the story, few days ago, on #webappsec (irc.freenode.org), Larholm told me about SWAAT a new (at least, for me) PHP SCSA (and not only PHP actually). At the time, I didn't have time to try it; but today, I took the time to compare PHP-SAT and SWAAT with a test which can be view as a quite-exhaustive-basic-flaw-checker (it means that there is maybe 6 different vulnerabilities with variants and false positive/true positive check implementation).
You can see the PHP test file here: tests.php

The result of the two runs can be find here: php-sat-test-output.phps and swaat-output.html
How to read the reports:

  • SWAAT: HTML file with table for each type of vulnerabilities, it will report multiple lines (each line is a vulnerability). If there is a /* fase */ in that line, then, this is a false positive.
  • PHP-SAT: PHP-SAT takes the PHP source code and transform it by adding some information. For the vulnerability report you will have to look for the Malicious Code Vulnerability (MCV). Other report are more quality oriented.


I will not spend time to explain the difference of the tools but the tools don't really have the same goal (even if we can use them for the same utilization). Well, with the default configuration of both tools, SWAAT is really better! But as for many Source Code Security Analyzers, the configuration is really important, so I would mitigate my conclusion on these tools, I really need to dive into the configuration of that two tools and redo the tests.

Monday, January 29 2007

When Black Box becomes Dark Box...

What if - in some cases - a Web Application Scanner (black box tester) could tell you that you have this vulnerability/weakness in your code at this line?

Got some ideas this morning on this, I'll try to implement this in Grabber in the next weeks.

Thursday, January 11 2007

What I want to for early 2oo7.

Even if i'll be busy with papers and tests, I really would like to do different things:

  1. Grabber: Adding an encoding stuffs for testing with different type of charsets (UTF-7/8/16 and other type of languages)
  2. Create a JavaScript functional analyzer: I've been thinking on this for a while, I think this is a good idea to detect XSS. I was thinking of using Stratego/XT for the parsing/AST construction; but still, because it's javascript, it's really hard to parse every possible things.
  3. XSS Handler: Just for fun, I want to do a PHP function for preventing XSS (using mb_strings) and the same kind of thing in Python

Tuesday, December 26 2006

Extended JavaScript/CSS parser for Grabber

I took some time this morning to extend the JavaScript parser; I also added a CSS parser.
The parsers can detect some basic comment based obfuscation. The JavaScript parser is also launched on the CSS's expression and url functions.
Now, I only have to parse the onbody, onmouseover etc. to get the JavaScript in these tags (samethings with the style tag for CSS); it should be done easily.

The following lines should be detected:

  • JS in CSS: background: expression(foo("script.ext"));
  • CSS: background: url(http://bad.com/script.jpg)
  • CSS: <style>

@imp/* This is a nested comment */ort('css-parser-nested-comment.css') </style>

  • JavaScript: foo(){var script="script.ext";var params="?kikoo=plop";async(script+params);}



The new Spider with the parser will be added in the next Grabber release.
I also have to say that I'm very proud that Simon Roses Femerling chooses to include the JavaScript/CSS parser from Grabber in the OWASP Pantera project :) !

Monday, December 11 2006

Grabber

I used to do craps with the Grabber distribution. Sometimes you could not use the soft at all...
It should be fixed now. The version seems to be quite "stable".

What's new in Grabber ?

  • Session ID / Server time retrieval
  • A better JavaScript/HTML parser
  • Some basic stuffs for the Hybrid Tool (I don't think I will support the regular expression soon even if it's almost in the code... I need to make my mind about the Source Code Scanner: I will look closer on PHP-Sat and other Stratego/XT based languages support)
  • A JavaScript quality checker plugin (using JavaScript Lint)

Friday, December 1 2006

Hybrid scanner / Crystal Ball

Description

Some days ago, I really start thinking about the hybridization of the web apps scanners.
I'm pretty sure this is the key since this kind of tools should be used not only by the security experts but the web developers.

So, at the question How a developer can really test his code there is lots of possible answer from classical test to source code analysis and application analysis. Because I'm a developer and I really want to have a tool that can do what I want and doesn't give me that much errors or "this page is susceptible to xss in the parameter..." when there can be lots of instance of this parameter in the code... I decided to add the feature to test the code then, with the results test the application itself. This is a crystal box tester/scanner.
The only real interest of this kind of tool is to decrease the number of false-positive.
I mean, you cannot say at all that a website is secure... but you can report some real vulnerabilities.


The creation process was the following (hum.. to much work in a day cause people - me - writing idiots descriptions).

Crystal Module Cooking Book

Make-ahead Tip: Prepare lots of coffee before starting...

Preparation: 24 hours

Ingredients:

  • PHP-Sat
  • Grabber Modules lambda

Tools:

  • Context editor
  • Python 2.4
  • Nice music (Opera is not needed but you should listen this)


Directions:

0/ Read the configuration file (with boolean operator in patterns)

1/ Scan the PHP sources with PHP-Sat handler (which copy everything in the '/local/crystal/' directory).

2/ Make a kind of diff then:

If the diff results, check for the patterns (given in the configuration file) Parse the PHP line under the end of the pattern Try to get a variable value

<after-hypothetical-stuff-toussa>
If no direct variable... backtrack sequentially or in the AST
</after-hypothetical-stuff-toussa>


3/ Generate the XML report of "the crystal-static-analysis" module
4/ Build a database of:

transformed_into_URL(hypothetical flawed files)  = [list of "flawed" params]


5/ Run the classical tools against the only parameters given buy the SwA tool

The results ?

Because I use Grabber to do the black box testing and PHP-Sat for the white box testing, the limitations is coming from the both sides.
First Grabber, because of the vector, the detection techniques and all flavors of Web Apps Scanners, and PHP-Sat... which is not finished yet and seems to only report the possible Cross-Site Scripting.

The configuration file

I'm quite proud of this, a good configuration file can make people happy :) to see that they can plug/do whatever they want with this. Let's see:

<?xml version="1.0"?>
<!-- Grabber configuration file -->
<crystal version="0.1">
  <!-- Give some information, distant/local files -->
  <url >http://192.168.1.2/bank</url>
  <files>\\192.168.1.2\htdocs\bank</files>
    <!-- Analyzer information, here PHP-Sat -->
    <analyzer>
      <path input="-i" output="-o">C:\msys\1.0\bin\php-sat.exe</path>
      <extension>php</extension>
      <!--
        Typical pattern block:
        
        11: not flagged php line
        12: /**
        13: 	pattern content...
        14: */
        15: php line with the flaw +- FLAGGED!
      -->
      <patterns start="/**" end="*/">
        <!-- Analyze with the pattern -->
        <pattern module="xss">
        PHP-SAT check (Malicious Code CodeVulnerability) __OR__ Pattern ID: MCV000
        </pattern>
        <pattern module="sql,bsql">
        PHP-SAT check (Malicious Code CodeVulnerability) __AND__ Pattern ID: MCV001
        </pattern>
      </patterns>
  </analyzer>
</crystal>

The configuration will support binary operators for combining some patterns, and later regular expressions in the XML.

Thursday, November 30 2006

I'm lazy but I do scan JavaScript

One of the biggest issue with Grabber and AJAX stuffs is to get the script names, parameters etc. The best solution is of course to emulate/plug a JavaScript interpreter in the scanner thus you can see what calls are running etc.

But it's hard to do (even if I plan to plug spidermonkey with Grabber...)! Then, I've made a very small JavaScript scanner that try to get the URL and the parameters of the scripts.
It seems to work well even if the list of theses "dumb_parameters" is, in my tests, not twice bigger as the real list; but it catch everything.

I should be able to say that it will run every callable servers scripts.

This will be in the next version of Grabber.

Tuesday, November 28 2006

Web Apps Scanner pre-release

Just an explanation about Grabber and a pre-release version.

I plan to release it when it will be really interesting, but you can reach it's page here: http://rgaucher.info/beta/grabber
But you can already download and use it. Every comments are welcome...

I <3 Bots!