Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Test Survey Results

by Anonymous Monk
on Aug 02, 2008 at 19:29 UTC ( [id://701844]=note: print w/replies, xml ) Need Help??


in reply to Test Survey Results

You have a bias that is showing. I laughed when I read "More than 10,000 tests" but nothing beyond "20 minutes or more". I was just running one test suite (of many) that has more than 10,000 tests (a few 100,000) which runs more than 5 hours. At my last job our full UT test suites ran several hours as well.

Replies are listed 'Best First'.
Re^2: Test Survey Results
by Ovid (Cardinal) on Aug 02, 2008 at 19:59 UTC

    I can assure you that many people who have extensive experience with testing will detect far more issues than just that (for example, test counts in xUnit versus TAP systems). I won't deny that outliers such as what you discuss are important, but I'm more interested in the common case right now. Outliers are generally handled differently because they are different.

    Cheers,
    Ovid

Re^2: Test Survey Results
by cog (Parson) on Aug 04, 2008 at 15:41 UTC
    :; time make test PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_h +arness(0, 'blib/lib', 'blib/arch')" t/*.t t/01-conversion_0_10000................ok t/02-conversion_10001_999999...........ok t/03-conversion_1000000_and_up.........ok t/04-special-cases.....................ok t/71-version_numbers_are_the_same......ok t/pod-coverage.........................ok t/pod..................................ok All tests successful. Files=7, Tests=10218, 7 wallclock secs ( 0.94 usr 0.15 sys + 6.70 c +usr 0.14 csys = 7.93 CPU) Result: PASS real 0m7.180s user 0m7.726s sys 0m0.324s

    There you go, over 10k tests in just 7 seconds.

    Granted, that is not a full system, just a module. However, my answers remain: my largest test suite has over 10000 tests, and it runs in just a few seconds.

    I also recall creating a system with ~400 tests that would run in just no time. Then again, the most enforced requisite of that platform was that it had to process over 1000 requests per second.

    And it did.

      Well...
      ambs@rachmaninoff ProjectoDicionario$ time make test
      PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0)" t/*.t
      t/empty_lines....ok                                                          
      t/last_line......ok                                                          
      t/test_terms.....ok                                                          
      All tests successful.
      Files=3, Tests=67546, 16 wallclock secs (13.43 cusr +  1.40 csys = 14.83 CPU)
      
      real	0m15.786s
      user	0m18.537s
      sys	0m1.806s
      

      This is not to show tests run quickly, as the tests are quite simple. What I would like to say is that these tests in particular are not testing code, but testing data quality (a PT dictionary formatting).

      Why is this helpful? Because when I too the time to write the tests, I found out that I had more than 6000 entries with formatting errors.

      Alberto Simões

      Obviously one can have a ton of trivial tests that run in very little time. That doesn't demonstrate a lack of bias. The bias was not contemplating the possibility that an extensive test suite could take much more than 20 minutes to run. In my experience, extensive test suites on a company-wide scale usually take quite a bit more than 20 minutes to run, so I don't consider such to be an 'outlier' as characterized by Ovid to dismiss the criticism. But I'd not have set the upper limit on number of tests at 10,000 either. I'm sure we each have over 100,000 tests at my current and previous employer.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://701844]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (5)
As of 2024-03-19 05:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found