http://qs321.pair.com?node_id=884537

andreas1234567 has asked for the wisdom of the Perl Monks concerning the following question:

Episode #285 of the Security Now! podcast discusses fuzz testing. Wikipedia defines it as:
Fuzz testing or fuzzing is a software testing technique that provides invalid, unexpected, or random data to the inputs of a program. If the program fails (for example, by crashing or failing built-in code assertions), the defects can be noted.
Perl has a strong history of automated tests, and I expected that perl and CPAN would have something to offer on fuzz testing, but a brief search revealed no relevant modules, tools or documentation. Does the nature of perl (i.e. being an interpreted language) make fuzz testing less relevant than e.g. web browsers or operating systems?
--
No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]

Replies are listed 'Best First'.
Re: Fuzz testing with perl
by Corion (Patriarch) on Jan 27, 2011 at 11:54 UTC

    Indeed, it would be interesting to fuzz-test the Perl interpreter by (Markov-)generating Perl code and parsing that code. But as this will only uncover bugs and memory leaks in the compiler part of Perl, there is little to be gained, security wise. If you are in the position to evaluate arbitrary Perl code, fuzzing isn't necessary anymore.

    Fuzz-testing (web) applications is a more promising avenue in my opinion, because there can well be oversights when handling input values in your program, even if you're running under taint mode. I've thought a bit about automating submission of generated form values and then hunting in the output for error messages or something that indicates that the application crashed or behaved unexpectedly. But as I didn't have a concrete application, and writing a general fuzzer was too complex, nothing came out of it. Writing the form submission is nearly trivial, but writing the evaluator that checks whether the result is as expected or deviates was what kept me from implementing something.

Re: Fuzz testing with perl
by marto (Cardinal) on Jan 27, 2011 at 12:06 UTC

    Metasploit was written in Perl but has since moved over to using Ruby. Fuzzy testing functionality was added at some point after this. If you intended to create a Perl module to do fuzzy testing I suppose you could first investigate how Metasploit is doing things.

Re: Fuzz testing with perl
by ELISHEVA (Prior) on Jan 27, 2011 at 12:08 UTC

    I'm not sure what you mean by a fuzz testing module. There are certainly plenty of random number/input generator modules on CPAN.

    However, I think a general purpose fuzz testing framework would be hard to develop for Perl applications. In the automated fuzz testing system I developed a few years back for a project (Java), I used reflection to discover parameters to constructors and methods and then generated random input to create objects or to pass to methods.

    Perl doesn't have a mechanism for retrieving the number or type of parameters of methods. The closest you come are prototypes, but they aren't widely used and they don't really specify parameter types so much as the context that should be used to interpret each parameter. You can get all of the subroutine symbols for a package, but you can't even tell which ones expect a class/object as a first parameter and which are normal functions.

    I suppose you could create such a system by slapping a layer of metadata and reflection on top of Perl (Moose anyone?), but then you aren't just programming Perl, you are effectively creating a new language.

Re: Fuzz testing with perl
by CountZero (Bishop) on Jan 27, 2011 at 18:38 UTC
    I am really surprised nobody mentioned Test::LectroTest.

    You define the conditions which must be fulfilled to validate your routines, then you define the realm of all input values and let Test::LectroTest test it.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

      Thanks for the pointer to LectroTest - I'd not come across that before. -- Richard Huxton
Re: Fuzz testing with perl
by jandrew (Chaplain) on Jan 27, 2011 at 16:45 UTC
Re: Fuzz testing with perl
by Gulliver (Monk) on Jan 27, 2011 at 15:24 UTC

    Are there non-interpreted languages with relevant modules, tools or documentation for fuzz testing?