nd
Subscribe to the RSS feed

Keyword - SAMATE

Entries feed - Comments feed

Tuesday, June 30 2009

NIST Static Analysis Tool Exposition special publication released

The NIST SAMATE project conducted the first Static Analysis Tool Exposition (SATE) in 2008 to advance research in static analysis tools that find security defects in source code. The main goals of SATE were to enable empirical research based on large test sets and to encourage improvement and speed adoption of tools. The exposition was planned to be an annual event.

SATE 2008 was one of my last project at NIST. I really enjoyed working on this project from the beginning, it was challenging especially because we had to create so many artifacts to make the tool reporting the weaknesses the same way, integrate them all together and provide ways for assessors to make meaningful reviews.

In a nutshell, we selected 6 different open-source programs (3 en C, 3 in Java) and made tool vendors running their tool on these test cases. Tool vendors were allowed to customize their tool if their tool provide such capability. Fortify was the only vendor who created a custom rule (to help the tool with a validation routine for MVNForum). Our goal was then to combine the results all together and analyze: provide information on the correctness of the tool.

If you are interested, you can download the SATE data and the NIST SATE Special Publication.

Thanks to all the SAMATE team for this effort, and especially Vadim Okun and Paul E. Black.

For more information, you can reach the SATE page at NIST.

Tuesday, September 23 2008

Last week at NIST

Every good things have an end... this is the time for me to leave NIST. So I will be a security consultant at Cigital, Inc..

I've been working at NIST for 2 years and a half as a Guest Researcher in the SAMATE Project. I originally came at NIST to do mostly statistical analysis or so, but it changed a lot! I started by building the SAMATE Reference Dataset website and this is how I started to learn about "security", but working with flawed source code. This was very obscure to me (I guess like everybody computer scientist specialized in applied mathematics) and I learned a lot about weaknesses, vulnerabilities, "how to find them?", scanners etc.

My first real security related work was about the Web Application Security Scanner Specification and then, design a way of testing the web apps scanners:

  • test suite with seeded vulnerabilities
  • checking the types of attacks
  • trying to explain the false-negative of the tools by a monitoring of what/where the scanner went in the application at a logical level, such as "did the tool logged in successfully? did it generate a couple of errors, did it try many times?

The goal of the 3 components based analysis is to really be able to understand what the tool is doing, if it didn't find a particular vulnerability, why?

One of the best moments I had at NIST was when we did the Static Analysis Tool Exposition. I was part of the organizers and from the beginning, it was a real challenge: choosing good test cases, criteria to evaluate the reports, etc. Of course, SATE 2008 was not perfect, we did many mistakes, but at least, we tried, we had some results and we learned a lot. I have good hopes for the next SATE, even though this is really challenging on many aspects:

  1. Not make people think/act like this is a competition (we sometimes see people claiming they won SATE 2008, but... well, there would be many things to say to them)
  2. Having a strong evaluation criteria (I guess this is challenging every time human assessment is part of the game)
  3. Solve the way to present data to the evaluators. We couldn't have the GUI of the tools etc. so our analysis (as an evaluator) was really limited and we sometimes had to guess what was the exact weakness report
  4. and finally, having more resources and help for evaluating the weaknesses reported by the tools (47k this year, one month to evaluate...)

Oh well, I will of course continue to follow what the SAMATE team is doing, even though I will be away and busy with other interesting stuff and I'm really looking forward to see the results of the current study we are running on the function-wise weakness characterization.

But for now, it's time for me to get some vacation, going back to France for almost one month, getting my worker visa etc.

Tuesday, June 10 2008

My talk at SAW: Automated Evaluation of source code analyzer output

It has been some time since I haven't post on my blog... well, I've been busy especially with the end of SATE, and oh well! had vacation :)

Anyway, at the next Static Analysis Workshop this Thursday, we're gonna talk about the SATE experiment and the observations/results we could get from this. I am then gonna talk about a tool I wrote in order to probe if a reported weakness is a false-positive: this is the Automated Evaluation.

The main idea of the Automated Evaluation, is to get some information on the source code and, under some assumptions, try to make a conclusion on the correctness of the piece of code. Behind all the reasoning from that particular tool, my approach had to be radically different than a classical SCA otherwise this would have been like creating a new SCA and this would have been obviously useless. The context of this automated evaluation is limited to the buffer overflows and this can only work for proving false-positive only!

So basically, I am reading the source code from the reported sink to the possibles sources and grabbing the actions that possibly affect the variable which have a role in the code.

These actions are like:

  • Allocation of a destination buffer
  • Computing the size of the source buffer(s)
  • Test for NULL
  • Test that involves the size of the buffers...
  • ... and some others

Then, once these actions are detected, the tool increments a global score of false-positiveness to this reported weakness. We then only have to set a threshold in order to know what correctness we want to have; this is really tied to the source code and how the program is developed.

Even though this evaluation method is not perfect, this was adapted to the C test cases we had in SATE 2008 since the global code quality was good. We can even say that the software were well written; it was then okay to make some assumption on the code such as:

  • If the size of the destination buffer is computed with the size of the source buffer, the size is good (basically: no off-by-one)

Also, the tool itself needs some information on the source code such since it uses regular expression to match the "actions"...



Here we are for a quick explanation and here are the slides: SAW: Automated Evaluation of SCA output

Wednesday, May 14 2008

Static Analysis Tool Exposition is over

Yeah, that's sad and also a relief: SATE is over. We actually released today the last stage of the evaluation (basically, the evaluation with some correction based on comments from the participants). Even though I would have prefer to have more feedback from participants on our evaluation, especially to increase its quality, I still think SATE is a good thing and will be an interesting resource for lost of researchers. This is, as far as I know, the only exhaustive resource on the subject (wild source code + weaknesses).

What do I want to do, see next? Since we have accumulated lots of data with the tool reports (raw weaknesses), the evaluations (I really want to thank MITRE's guys, especially Steve Christey and Bob Schmeichel for their help), I'm looking forward to do data analysis and trying to extract some limited results on it.

Anyway, this was overall a good experience, I actually did my first real code review mostly on lighttpd, dspace, mvnform and naim, I think I know way more on how detecting vulnerabilities, I also have been asking myself about how to rate vulnerabilities such as Cross-Site Scripting (hopefully, I will release the little document I wrote about it), I learned so much about how people are writing code trying to understand the design, the code etc. in the applications.

Also, hopefully, I will be able to release the website I developed to handle the weaknesses from different tools. It is, I think, interesting if you are working with more than one assessor. You can send evaluation, comments, merging the weaknesses etc. with a web interface. Even though it needs improvements (it has been done in less than 2 weeks) I think this would be an interesting piece of software for people who are dealing with tons of weaknesses. Another interesting point is that we (at NIST) may open that website for everybody in order to make new evaluation in order to increase the quality of the data we currently have.

Oh well, it seems like a journey is really close to its end, it was such a good time sometimes, and some other time such consuming work. We've been dealing with fifty thousands of weaknesses, dozen of tool reports, and almost tens of test cases... I will keep you posted about the next decision we are gonna make with SATE and hope that lots of people will find in this "exposition" the most they could get.

Thursday, February 14 2008

SATE ready to go + weaknesses walker + Shmoo + 100

Tomorrow will start SATE 2008: the registered participants will be able to get the test cases associated to the tracks they want to participate in. They will have until the 29th of February to send the report of the tools. We are all pretty excited here before the start. It was a real rush for finding the test cases that we think are good for such an event...

Anyway, just a news to release a python script which is definitely SATE oriented. The idea is only to convert the output of some free tools into the SATE XML format. The script is handling Flawfinder, ITS4 and RATS. It can also look at the NVD for the product and the version in order to retrieve the known vulnerabilities.

You can download the script weaknesses walker as a zip file or just the python script (you will need wwwCall for the NVD scrapping part; wwwCall is also included in the zip).

Example how to use ww with flawfinder:

./ww.py --tool flawfinder --file myproject.out.xml --format sate /home/romain/myproject

or for the NVD scrapper:

./ww.py --vdb winamp 5.2 --file winamp_5.2.nvd.xml

For the next version of ww, I may add the possiblity to play with the SATE XML format itself, such as merging the results of different tools with comparison of report or even just the report of multiple tools...

Also, if you are coming downtown DC this weekend for ShmooCon or even BlackHat DC 2008, if you wanna have a beer just drop me a mail. I wasn't able to find a ticket for Shmoo so will not go, but I will meet with dre and marcin from ts/sci security... so if you are around, just tell me I would be happy to meet more sec. people

The last thing is that this post is my number 100!

Tuesday, February 5 2008

NIST Static Analysis Tool Exposition: No, this is not a competition!

I've was happy yesterday when I learned that Fortify will participate to the Static Analysis Tool Exposition (SATE) we are currently organizing. And even more when I saw this morning Brian Chess blogging about SATE.

We've been working on SATE since our last Static Analysis Summit and, helped with a couple of existing exposition already existing at NIST such as TREC etc. for the guidelines, the rules and so on. But even so, we had some example, we had three difficult tasks:

  1. Make people agree on the fact that it is not a competition
  2. Make vendors participating (if you are a vendor, reading this please, subscribe for participating at SATE)
  3. Choosing the test cases

The last point is not solved yet, and even, none of them can be considered as solved since not everybody is participating to the 2008 exposition (which has 2 tracks: C and Java), but we've been seeking for good test cases in C and Java. Good test cases... means not too big, not too small and having exploitable vulnerabilities. By the way, if any of the readers of this blog have some idea of Java or C test cases that would be good test cases, please, send me links, ideas or whatever :)

Anyway, SATE is on his way, I hope more tool makers will sign up for participating at this experiment.

Maybe another point, due to my usual blogging on web security and web apps security scanners, if SATE is a success as we expect it to be, we may open new tracks for... web application security scanners and I would love to have special tracks for security metrics (I want to show up!! :p)

Thursday, January 31 2008

Talk: Problems and solutions for testing web application security scanners

I just came back from the DHS Forum on the Software Assurance where I gave a talk about testing web application security scanners, and especially, the problems and some solutions for testing the scanners.

The presentation is an introduction to a methodology I've been developing at NIST for a while now. This presentation is the follow-up of the Verify Conference slides and also the talk I gave at HICSS (I will release the slides from this presentation when engadget.com will fix the vulnerabilities that I used in order to show the different variation of attacks for introducing the level of defenses)

You can reach the DHS Forum slides as a Google presentation.

Sunday, December 2 2007

Yet another study oriented release

I've been working a couple of months on a project named php-ast/oracle. I am opening the source of the project today because I think that people may be interested in such a code. Roughly, php-ast/oracle is able to get/transform information on a php source code, I used it for: creating real obfuscations (control-flow, data-flow), implementing security metrics, writing a converter from php to c++ for static analysis purpose and some other stuff such as variables flow etc.. You can have more information here: http://trac2.assembla.com/php-ast. I may post about this project later don't have much time now...

But this news is only for releasing a script I used a lot this last weeks; a PHP preprocessor. I've been using this preprocessor in order to clean the crappy PHP code we can found in the wild... in order to use php-ast/oracle correctly for calculating security metrics and so on.

The preprocessor is actually doing 3 things:

  • Simplifying the strings (keeping only the php variables in the strings -- really important for keeping the AST small with SQL queries and so on, because the strings could be evaluated in PHP, the AST would need to tokenize the strings)
  • Removing comments and HTML
  • Resolving the file inclusions (not for dynamic variable inclusion of course, but it's working with define names and static names)

The preprocessor is available here: preproc.zip

Thursday, November 1 2007

My talk at the Verify Conference

Last Tuesday, I went to the Verify conference to give a talk about Web application scanners evaluation: what we are actually doing at NIST. I'm gonna make a simple entry reviewing what I actually talked about. The slides are here.

First of all, the evaluation was made with a test suite I made. The choices for the test suite are kinda simple, I wanted something really close to a real website. So I decided to use a real website (not a couple of test cases). The website contains multiple seeded vulnerabilities from different kinds (XSS, SQLi, RFi, CSRF, etc.). The website is actually configurable in a sense of vulnerability: you can choose what vulnerabilities will be in the website or not (let's say, I only want to have XSS vulnerabilities). Moreover, in order to see the web apps scanners capabilities, we can select a type of defense for the current protection: the level of defense.

Level of defenses

Programmers are different. They have different background, knowledge and approach to solve security problems. The filters we can see in wild web applications are not equivalents, some are good, some are just bad and we have the full shade of effectiveness. So, in order to test web apps scanner with different difficulties (for them) we implemented different level of protection around the vulnerabilities: the level of defenses.

A simple example: SQL Injection

  • Level 0: No protection
  • Level 1: Typecasting (in order to convert integer, boolean, double, strings, dates etc.). This protection will limit the SQL Injection on SQL native number types (integers will be converted as integer: 1' OR 1=1-- will be converted into 1).
  • Level 2: Escaping the meta-characters. We are protecting about quote injection, etc.
  • Level 3: Hiding the MySQL errors, we will now have Blind SQL Injections.
  • Level 4: Restricted user management.
  • Level 5: Using prepared statements.

Since the level of defenses will be use in combination, the order is important. (combination: level 2 = level 2(level 1(level 0))). So, using these level of defenses we are able to select the difficulty that the tool will have to break the vulnerabilities. For the results, if you are looking at the slides, in the detection rate slide, you'll see that there is not result for the level 2 which means that no tools were able to find vulnerabilities in the level of defense 2.

Attack Surface Coverage

Another point I have been working on is the attack surface coverage. A webapps scanner is not a simple piece of software which launches attacks! The crawling/parsing step is actually really important maybe the most important since it will try to understand the application. The attack surface of the test application is the places where the user has a direct interaction, means no algorithms etc. just inputs handling, error messages etc.

Here is an example of attack surface coverage check points (with numbers) for a login function:

(1) Touch the file [login.php]
if ( all fields are set ) then
	(2) All fields are set [login.php]
	Boolean goodCredentials = checkThisUser(fields)
	if ( goodCredentials ) then
		(3) Credentials are correct; the User is now log in [login.php]
		registerCurrentUser()
	else
		if ( available login test > 0 ) then
			(4) Login information incorrect [login.php]
			displayErrorLogin()
			available login test -= 1
		else
			(5) Too much try with wrong credential [login.php]
			displayErrorLogin()
			askUserToSolveCAPTCHA()
		endif
	endif
endif

Basically, we would like the scanner to use the normal behavior paths and also the abnormals (errors etc.) in order to find vulnerabilities there such as Information Leakage etc. Just a note about the attack surface coverage rate: this number cannot be interpreted alone. You need to use this with the detection rate and the false positive rate. In the slides you can see that the tool A as a 25% attack surface coverage of the application, but this is also the tool with best findings and no false positive. This means that the tool were able to find 33% of vulnerabilities (best results from all the 4 scanner we tested) in 25% of the application which can be considered as accurate compared to the others.

The attack surface coverage may have an important impact, depending on what type of testing you are doing with your webapps scanner. If you want a tool to run at the end, doing a full assessement, then you will need a tool which as a very good coverage (since you only rely on that). But if you are looking for a tool which is fully integrated in your testing process (testing == quality and security) then, I think it's better to have an accurate tool which will cover a lower surface, but the tool will cover the important points.

Conclusions

This is actually hard to make a real strong conclusion about the results given in the slides. The test application is a real simple website (banking application) and is far from a real company website; this is a huge confounding factor. Another problem is that I did the evaluation one vulnerability at the time (and one level of defense at the time). This prevent a couple of real life behaviors...

Tuesday, October 16 2007

Stuck at data-flow? Do box-modeling!

Since yesterday, I'm working on a data-flow problem. I need to model a function and I should do all the data-flow process. Well, that's kinda long if I have to do that on all functions and especially I will never use much of the information I would generate by analyzing the tree associated to the function (local variables etc.). So what the point of doing that? None.

I was stuck at this point, didn't find a good way to model a function (entry parameters, global calls etc.) so I thought of reasoning as a crystal ball. I can see what it is, but it's kinda blurry :) I am now modeling a function as inputs and outputs, only in terms of functions and global variables interaction. By this, I should be able to see the possible interaction of the given function on the system. Hope it's gonna work well!

Wednesday, October 10 2007

Working around security metrics...

I'm not gonna write a long entry about Security Metrics, but since I've been working on this for a couple of weeks now, I have some thoughts. Evaluating the security of a source code is actually pretty hard. Even if I'm sure there is a lot of source code security metrics out there, it's often (I guess) hard to compute. Basically, you will need to know lots of things about the source code then, you need an engine working on the AST , data-flow etc.

This is what I've done for a couple of months, an engine which is working on XML AST, generated by yaxx (this is the same engine that I use to do source code modifications, obfuscations, etc.).

With Vadim Okun, we had the idea of computing the "size" of the security in a source code. The idea is pretty simple and we are aware that this is limited to implementation flaws and not design flaws for now. The "size" of the security is the number of inputs going to sinks.

The inputs have to be taken in the large sense, these are in fact all the variable that are derivate from direct inputs. Here is a simple example of the variable diffusion:

$a = $_GET['foo'];
$b = htmlentities($a);
echo $b;

We are here counting $a and $b since $b is a modification of $a which is a direct input. We are using the same methodologies for all possible modification (concatenation, cast, etc.).

Once we know these variables, we are counting the ones that are going to sinks. The sinks are a list of function such as 'echo', 'mysql_query', 'fopen', and so on. Our list of sinks is directly coming from the PHP-SAT project. In the previous example, the metric result is 1 since there is only one sink 'echo' where a derivate input is going to.

And here we are, this is a fairly simple (in the idea, not the implementation) way to evaluate the possible security problems that you can have in your source code. We are going to try and evaluate this metric on different open source project (wordpress, joomla, mediawiki etc.). I'm sure this is really incomplete: first because we are only counting the security problems that are coming from inputs but also because it really depends on the programmer (his style of programming).

An other example is available here: smetric.pdf

Next Improvements

For the revised version, the first add would be to count the output validation problems. But for that purpose, I need a stronger data-flow analysis which would analyze in function definitions (not done yet). Then, I will be able to trace everything coming from supposed secure sources (databases, resources, local files, etc.) to sinks. Maybe the weight of such flows would be different than the first one (input to sink)...

Tuesday, September 4 2007

Source Code Obfuscation

Source Code Obfuscation is actually a powerful tool for testers. Whether you use it to obfuscate your bytecode (Java, .NET etc.) or increasing the code complexity of your current source code.

Working at SAMATE we are also playing, tweaking, testing, stressing source code analyzers. And now you see the relation. I'm writing a source code obfuscater in order to increase the complexity of our test cases and see if the tools are still doing well.

Thus, I was able (with good documentation, and yaxx) to create one. It currently only add control flow complexity (and of course renaming classes, functions and variables).

Some words on obfuscation

You may have heard about obfuscation in a sense of making the code unreadable for users. This is not what I'm interested in. I want to modify the actually source code, adding some information in it, some tests... I need the outputs of the original program and the obfuscated one to be the same, otherwise we cannot consider the source code as being equivalent.

So for example if I do:

if (var == 0) { 
  echo 0;
}

I will have the same behavior with this source code:

x = some_value;
if (var == 0 or x*x < 0) {
  echo 0;
}

Even though they have the same output, the second one is more complicated since it adds one other test.

Test case wise example

To see what it exactly does given this original source code

<?php
	$b = 0;
	$c = "Salut";
	$len = strlen($c);

	function fct($a) {
		return $a . "_1\n";
	}
	
	class T {
		function foo() {
			echo "test\n";
		}
	}
	echo fct(0);
	echo T::foo();
?>

...and by choosing one of my control flow obfuscating pattern

<?php
	class rand_class_name {
		function rand_func_name_2($rand_name_1) {
			return $rand_name_1 + 1;
		}
	}

	function rand_func_name_1($rand_name_2) {
		return $rand_name_2 + 1;
	}
	
	if (rand_func_name_1(0) > 0 && rand_class_name::rand_func_name_2(0)) {
		$enter_the_new_statement;
	}
?>

To have this result:

<?php
	function HXvE5Plwxp0RSoQM ( $ZMfP98Az96Rq67j6 ) {
		return $ZMfP98Az96Rq67j6 + 1 ;
	}
	class TF03COvMuzXRQcCK {
		function Ltghf3a0McCI8RaZ ( $V309os5vQo15ak9b ) {
			return $V309os5vQo15ak9b + 1 ;
		}
	}
	$b = 0 ;
	$c = "Salut" ;
	$len = strlen ( $c ) ;
	function fct ( $a ) {
		return $a . "_1\n" ;
	}
	class T {
		function foo ( ) {
			echo "test\n" ;
		}
	}
	if ( HXvE5Plwxp0RSoQM ( 0 ) > 0 && TF03COvMuzXRQcCK :: Ltghf3a0McCI8RaZ ( 0 ) ) {
		echo fct ( 0 ) ;
	}
	if ( HXvE5Plwxp0RSoQM ( 0 ) > 0 && TF03COvMuzXRQcCK :: Ltghf3a0McCI8RaZ ( 0 ) ) {
		echo T :: foo ( ) ;
	}

?>

How it actually works

First of all, the engine only works on Abstract Syntax Tree (AST) in order to do powerful manipulation and code refactoring. The idea is to take a couple of transformation patterns (the second source code is in fact a complicated one), and fitting this patterns with the original source code.

The patterns are meta code. You can see that they are in PHP using some names such as $rand_name_1 etc. this means that the engine will generate one unique name for each of them and replace it before the actual refactoring.

Select what I want to obfuscate is not a real problem, but for now I only selected the top statements and will apply the whole modifications to each of them.

A little schema explaining a little how it works is available here: schema_obfuscation.png

What's next

The applied control flow obfuscating pattern is on of the many I do have for now (many more to come), and I guess this is kinda promising, lots of interesting studies should come now.

Currently the tools is only for PHP but I should make it general by using my own AST nodes names and then be able to do code transformation on C, C++, Java etc.

There is no release of the tool (written in C++) right now, I will wait until it's more than correct and clean. I also need to do data obfuscation (using indirections etc.). The program will of course be public and free for everybody when it's gonna be ready.

Wednesday, August 29 2007

I now understand why it's difficult!

Okay, I know for the halting problem etc. Some theoretical stuff... But now that I'm working on one, I have to say:

Damn! That so complicated to do a source code scanner!

The dataflow is a real pain in the ass, and we know that it's impossible to have a real and full dataflow. But well, we need to do some. The dataflow is more complicated theoretically but what about the control flow? No really easier! I mean... that's easier but there are so many things to understand, so many patterns to recognize in order to build the model of the source code... And I'm not even talking about inter procedural stuff, multi-file source code etc.

So, I'd like to apologize to "I don't remember who are these people" but some source code scanners are good :) Well... for the moment! I'm really waiting for to see more high-tech stuff and AI in these kind of programs...

Anyway, I'm currently building a core engine working on a AST tree generated by yaxx (XML version). I have two short terms targets:

  • Real Obfuscation (from one source code to an equivalent with a different control flow... yes, not only rename the variables, functions, classes etc.)
  • A variable tracer (tool for pen-tester: $_GET['foo'] -> ($foo <- htmlentities()) -> echo or this kind of stack...)

Thursday, August 23 2007

Web App Security Scanner Evaluation Criteria

Here is a new interesting project: WASSEC. This WASC's project is run by Anurag Agarwal and is about the evaluation of web application scanners such as Watchfire's AppScan, SPI's WebInspect etc.

If you are in the field, don't wait to help us :). Here is Anurag's words:

Thank you all for your patience. We have received an overwhelming response from the WASSEC (Web Application Security Scanner Evaluation Criteria) project. To proceed with the project please

1. Please email wasc-wassec-subscribe(AT)webappsec(DOT)org and reply to confirmation email.

2. It is moderated subscription so every contributor has to be approved to send messages to the list.

3. Once you are subscribed to the list, then email wasc-wassec(AT)webappsec(DOT)org to post messages.

All further communication will be done through the mailing list. Please keep checking your junk mail folder in case some messages might go there. We are also in the process of setting up a wiki for the length of the project to post updates, etc. Until then I will be updating my blog with the project details.

Once again, thank you for your participation.

You can checkout the project here: http://webappsec.org/projects/wassec

Wednesday, February 7 2007

How you should design a test suite for Web Apps Scanners

If you have ever think about using a web application scanner for testing the security of your website, you certainly made a choice: Which web apps scanner should I buy/use ?

In this post, I will not tell you what is the better black box tester for whatever kind of web application.

The web applications may be very different, the tools are different and thus they could have different efficiency (i think it's non countable noun)... If you read this, you probably know that I am talking about scanners such as WebInspect, AppScan, Acunetix, Hailstorm, Pantera, Grabber... In the following sections, I will explain a main idea that should be used for testing such a tool.

A test suite for our tools is a website, this website has typically vulnerabilities; you can see this kind of website by watchfire, spi but also WebGoat, SiteGenerator or others. But all of these websites are not realistic and do not consider that the vulnerabilities may exist in different instances: variants.
If you don't know what is a variant check at the XSS Cheat Sheet (RSnake) or at the Attack Patterns (Sean Barnum). A variant is what the hacker use to perform his exploit.
A simple example for XSS is: Let's say you protect your website against XSS by checking the <script> tag; this is not perfect and not good because there is some way to insert other type of XSS strings (onmouseover).

Here comes the concept of Level Of Defense. If you are a developer you think about filters, if you are an attacker you think about variants of vulnerability and attack patterns. The level of defense of a website is the strength of its filters again a given vulnerability.

For a SQL Injection you can have multiple type of filters... Here is a possible list of levels for the SQL Injection:

  • Level 0: Show SQL errors / No input filtering
  • Level 1: Hide SQL errors / No input filtering
  • Level 2: Typecasting (integer, string etc.)
  • Level 3: Escaping input strings
  • Level 4: Restricted accounts...
  • ...


In the concept of the level of defense, it's important to not that depending of the type of vulnerability (weakness, failure...) the level n-1 is also performed in the level n or the level n is stronger (for the same variants) than the level n-1 (for instance, for Weak Hash Function it's not possible but using SHA-1 instead of MD5 is a level of defense higher).

A Key point: When you are implementing a level of defense for a vulnerbility, you must be sure that your implementation does the whole thing for that type of filter. For example, if you are escaping the HTML entities, you need to do all not only '<', '>' and in the next LoD escaping ' and ".

Why is the level of defense better than a simple system with vulnerabilities?

With the level of defense, you can calibrate a type of website which may be close to yours; you can construct a test suite with your kind of level of defense and see how the tool detect the vulnerabilities when the LoD increase. It is also a good way to know the state of the art of the tools for detecting vulnerabilities...

The idea was developed to create a test suite in order to evaluate web apps scanners; in this test suite we can select the current type of vulnerability and its level of defense (the hardness to break):


Saturday, February 3 2007

pyIndex: File Indexer in Python

A couple of months ago, I had to make a Source Code Search Engine for the SAMATE Reference Dataset. The organization of our source code are not really common but still, it's easy to understand and organized.
I now release this tiny python script in the beta section: pyIndex.

You should have all the information you need to use/adapt this script for your own purposes; it uses a MySQL database and MySQLdb to connect to the database. The script is only for adding some words or references in the database, the search is not done (but it's only a really simple SQL query...)

Wednesday, January 10 2007

Test Suites for Web Application Scanners

For a while, I've been working on a test suite for evaluating web application scanners. Now I have a test suite (PHP/MySQL/AJAX) with a bunch of variable vulnerabilities:


But there is a problem for a full evaluation. Web Application are not only a simple schema of scripts and databases and complex relation, there is also server configuration, infrastructure, different type of databases etc. Thus, I really have to create different test suites for a good coverage of what web apps could be.
I plan to use:

  • Ruby On Rails framework
  • ASP.NET/MS SQL based application
  • JSP application


This should cover the differnt type of application but I still have to think about server types, architectures,multiple databases etc.

Tuesday, November 28 2006

Web Apps Scanner pre-release

Just an explanation about Grabber and a pre-release version.

I plan to release it when it will be really interesting, but you can reach it's page here: http://rgaucher.info/beta/grabber
But you can already download and use it. Every comments are welcome...

Friday, November 17 2006

Tools evaluation state...

For my work in the Samate Project, in the web apps scanners evaluation, I made a website with a variable level of security, because I was totally not satisfied by the Watchfire or the SPI-Dynamics demo websites.

Then, I started to consider this website as a test suite... The problem is the gap that could be in different type of tools: - basic tools (Paros, Pantera, Wapiti etc.) - famous commercials (webinspect, appscan, ntospider etc.) Mainly because of the AJAX.

Actually I use AJAX in different part of the website such as login system, registration, dynamic verification and I'm sure that if you cannot interpret the JavaScript, you cannot see the vulnerabilities in this code. Maybe the tools can parse some urls... maybe i have to create another "more classical" website, with only {php,mysql,sessions,cookies}... Wait and see the first results

Thursday, November 2 2006

Gatcha!

Because I don't really want to increase my IDS points (currently 15000) at NIST, I have to setup a website for doing my web apps scanners evaluation on a local network. It's pretty annoying because I could not test some tools such as AppScan or WebInspect since I only have the trial version.

Anyway, I try to make a website with a variable degree of security from the weakest to the more realistic one. I also use AJAX on limited parts to see how tools can handle this...

By the way, I will need to read lots of papers, forums, techniques about "how to avoid the vulnerability v"... I mean efficient techniques.

Monday, October 30 2006

Wapiti! Piti piti

You: What a sense of humour!
Me: I know

By the way, this thread is only to give a URL: http://wapiti.sourceforge.net This is a quite simple web apps scanner. I have to test it for wednesday (when I'll give a presentation on Web Apps Scanners with Demos).

And because I'm glad you're reading theses lines, here is a new 'stuff' on OWASP website (meaning Cool Stuff For Web Developpers/Security) : Pantera

I <3 Bots!